A Study of BFLOAT16 for Deep Learning Training
Open accessAbstract
This paper presents the first comprehensive empirical study demonstrating the efficacy of the Brain Floating Point (BFLOAT16) half-precision format for Deep Learning training across image classification, speech recognition, language modeling, generative networks and industrial recommendation systems. BFLOAT16 is attractive for Deep Learning training for two reasons: the range of values it can represent is the same as that of IEEE 754 floating-point format (FP32) and conversion to/from FP32 is simple. Maintaining the same range as FP32 is important to ensure that no hyper-parameter tuning is required for convergence; e.g., IEEE 754 compliant half-precision floating point (FP16) requires hyper-parameter tuning. In this paper, we discuss the flow of tensors and various key operations in mixed precision training, and delve into details of operations, such as the rounding modes for converting FP32 tensors to BFLOAT16. We have implemented a method to emulate BFLOAT16 operations in Tensorflow, Caffe2, IntelCaffe, and Neon for our experiments. Our results show that deep learning training using BFLOAT16 tensors achieves the same state-of-the-art (SOTA) results across domains as FP32 tensors in the same number of iterations and with no changes to hyper-parameters.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Bfloat16 precision offers significant speedups for BERT models while maintaining comparable accuracy to FP32.
“This precision offers significant speedups for BERT models while maintaining comparable accuracy to FP32.”
From A Survey of Techniques for Optimizing Transformer Inference (2023) · cites this paper as [164]
Section 4.2.2 (p. 5) reports BFLOAT16 accuracy matching or exceeding FP32 for a neural machine translation model. Section 2 (p. 3) reports speedups for other TensorFlow tasks, but our copy does not report BERT results or establish significant speedups for BERT. (LLM, conf=0.88)
Experimental — accuracy not yet measuredAll the discussed numerical formats offer several benefits over traditional floating-point precision, enabling faster and more efficient inference with reduced memory bandwidth.
“All these formats discussed here offer several benefits over the traditional floating-point precision, enabling faster and more efficient inference with reduced memory bandwidth.”
From A Survey of Techniques for Optimizing Transformer Inference (2023) · cites this paper as [164]
The Introduction (p. 1) links low-precision computing to faster deep-learning computation and says BFLOAT16 reduces communication volume in distributed training. It mentions mixed-precision inference, but does not establish that all the formats discussed make inference faster or reduce its memory bandwidth. (LLM, conf=0.85)
Experimental — accuracy not yet measuredEmerging numerical formats such as Bfloat16, MSFP, Tensorfloat32, and FP8 provide better computational efficiency and accuracy for several transformer workloads.
“The emerging numerical formats such as Bfloat16 [164], Microsoft Floating Point (MSFP) [165], Tensorfloat32 [166], and FP8 [167, 168] provide better computational efficiency and accuracy for several transformer workloads.”
From A Survey of Techniques for Optimizing Transformer Inference (2023) · cites this paper as [164]
The BFLOAT16 citation [164] supports only the BFLOAT16 part of the sentence. Section 3 (p. 3) describes potential hardware area and power savings, and Section 4.2.2 (p. 5) reports equal or better translation accuracy than FP32 for a recurrent model. The paper does not establish improved computational efficiency and accuracy for transformer workloads; its evaluated translation model uses LSTMs, not a transformer. (LLM, conf=0.88)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · cited 5× · see citing works
“When training ResNet-50 on Imagenet using the AVX512BF16 instructions, we achieved a Top-1 accuracy of 75.62%, which matches the current state-of-the-art performance.”
MethodInfluentialImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155 · cited 5× · see citing works
“Here we choose AlexNet (ILSVRC 2012) [22] and ResNet-50 (ILSVRC 2015) [18] as representative models for the BFLOAT16 evaluation.”
MethodInfluentialVery Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 74,248 · cited 1× · see citing works
“The network uses a “VGG loss” function using a pre-trained VGG19 network [33].”
MethodImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333 · cited 1× · see citing works
“However, all the aforementioned methods are shown to work for smaller benchmark model/data-sets only and invariably result in a non-trivial drop in accuracy with larger ImageNet data-set [9] and classification task [32].”
MethodImageNet Large Scale Visual Recognition Challenge
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. (2015) · International Journal of Computer Vision · doi:10.1007/s11263-015-0816-y · Cited by 39,380 · cited 1× · see citing works
“However, all the aforementioned methods are shown to work for smaller benchmark model/data-sets only and invariably result in a non-trivial drop in accuracy with larger ImageNet data-set [9] and classification task [32].”
MethodPhoto-Realistic Single Image Super-Resolution Using a Generative Adversarial Network
Ledig, C., Theis, L., Huszar, F., Caballero, J., Cunningham, A., Acosta, A., Aitken, A., Tejani, A., Totz, J., Wang, Z., et al. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.19 · Cited by 12,696 · see citing works
Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network
Ledig, C., Theis, L., Huszár, F., Caballero, J., Cunningham, A., Acosta, A., Aitken, A., Tejani, A., Totz, J., Wang, Z.-H., et al. (2016) · Computer Vision and Pattern Recognition · doi:10.48550/arxiv.1609.04802 · Cited by 12,624 · cited 5×
“Using multiple deep learning frameworks modified to insert appropriate Quantlib calls, we provide SOTA results for: AlexNet [22], ResNet-50 [18], DC-GAN [29], SR-GAN [24], DeepSpeech2 [4], GNMT [37], and two industrial workloads namely: a Deep and Cross Network, and a DNN Recommendation System.”
BackgroundMethodInfluentialUnsupervised representation learning with deep convolutional generative adversarial networks
Radford, A., Metz, L., & Chintala, S. (2015) · arXiv · doi:10.48550/arxiv.1511.06434 · Cited by 12,235 · cited 7× · see citing works
“We have implemented DC-GAN in Caffe and for our experiments all the input tensors (activations, weights) are converted to BFLOAT16 for convolution layers (in both the generator and the discriminator), while only the input activations are converted to BFLOAT16 for batch normalization layers; all other tensors are maint…”
BackgroundMethodInfluentialTensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems
Abadi, M., Agarwal, A., Barham, P., Brevdo, E., Chen, Z., Citro, C., Corrado, G. S., Davis, A., Dean, J., Devin, M., et al. (2016) · arXiv · doi:10.48550/arxiv.1603.04467 · Cited by 9,408 · cited 2× · see citing works
“There are at least three half-precision formats in the domain of mixed precision training of large neural networks: FP16 [28], 16-bit Integer based [9], and BFLOAT16 [10, 4, 3].”
MethodJulia: A Fresh Approach to Numerical ComputingPreprint
Bezanson, J., Edelman, A., Karpinski, S., & Shah, V. B. (2014) · SIAM Review · doi:10.48550/arxiv.1411.1607 · Cited by 7,077 · cited 1× · see citing works
“The Julia language [5], which is designed to provide high-performance without sacrificing ease of programming, has also been shown to be benefited from BFLOAT16 [11].”
Background“Librispeech: an ASR corpus based on public domain audio books,”
Panayotov, V., Chen, G., Povey, D., & Khudanpur, S. (2015) · 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2015.7178964 · Cited by 6,072 · see citing works
Graves, A., Fernández, S., Gomez, F., & Schmidhuber, J. (2006) · Proceedings of the 23rd international conference on Machine learning - ICML '06 · doi:10.1145/1143844.1143891 · Cited by 5,370 · see citing works
Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
Wu, Y., Schuster, M., Chen, Z., Le, Q. V., Norouzi, M., Macherey, W., Krikun, M., Cao, Y., Gao, Q., Macherey, K., et al. (2016) · arXiv · doi:10.48550/arxiv.1609.08144 · Cited by 5,319 · cited 5× · see citing works
“Google’s Neural Machine Translation (GNMT) is the SOTA neural machine translation model using a recurrent network.”
MethodInfluentialHan, S., Mao, H., & Dally, W. J. (2015) · arXiv · doi:10.48550/arxiv.1510.00149 · Cited by 4,662 · cited 1× · see citing works
“Inference has witnessed a proliferation of mixed precision compute [19, 33, 27, 25] where different operations execute at different precision, all the way from binary/ternary operands to 16b floating point.”
BackgroundXNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networks
Rastegari, M., Ordonez, V., Redmon, J., & Farhadi, A. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46493-0_32 · Cited by 4,408 · cited 2× · see citing works
“Inference has witnessed a proliferation of mixed precision compute [19, 33, 27, 25] where different operations execute at different precision, all the way from binary/ternary operands to 16b floating point.”
BackgroundMethodBinarized Neural NetworksPreprint
Hubara, I., Soudry, D., & El-Yaniv, R. (2016) · arXiv.org · doi:10.48550/arxiv.1602.02505 · Cited by 2,241 · see citing works
Quantized Neural Networks: Training Neural Networks with Low Precision Weights and ActivationsPreprint
Hubara, I., Courbariaux, M., Soudry, D., El-Yaniv, R., & Bengio, Y. (2016) · Journal of machine learning research · doi:10.48550/arxiv.1609.07061 · Cited by 2,196 · cited 2× · see citing works
“Literature shows that various different reduced precision data representations have been investigated which can be broadly classified into two types: the more standard floating-point based formats [28, 15, 12] and custom fixed point based formats [37, 7, 18, 22, 23].”
BackgroundMethodDeep Speech 2: End-to-End Speech Recognition in English and MandarinPreprint
Amodei, D., Anubhai, R., Battenberg, E., Case, C. T., Casper, J., Catanzaro, B., Chen, J., Chrzanowski, M., Coates, A., Diamos, G., et al. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1512.02595 · Cited by 2,191 · see citing works
BinaryConnect: Training Deep Neural Networks with binary weights during propagations
Courbariaux, M., Bengio, Y., & David, J.-P. (2015) · arXiv · doi:10.48550/arxiv.1511.00363 · Cited by 2,180 · cited 1× · see citing works
“This idea is further advanced in [7] where the authors showcase training with only binary weights while keeping all other tensors and operations in full precision.”
BackgroundDeep & Cross Network for Ad Click Predictions
Wang, R., Fu, B., Fu, G., & Wang, M. (2017) · ADKDD@KDD · doi:10.1145/3124749.3124754 · Cited by 1,625 · cited 2× · see citing works
“The accuracy of the recommendation system models is measured by the log loss [38, 43, 29], which predicts when the users will click on ads.”
MethodDeep Learning with Limited Numerical Precision
Gupta, S., Agrawal, A., Gopalakrishnan, K., & Narayanan, P. (2015) · arXiv · doi:10.48550/arxiv.1502.02551 · Cited by 1,364 · cited 3× · see citing works
“Numerous studies have shown that 16-bits of precision is sufficient for training deep neural networks[16],[26],[21],[8].”
BackgroundMicikevicius, P., Narang, S., Alben, J., Diamos, G., Elsen, E., Garcia, D., Ginsburg, B., Houston, M., Kuchaiev, O., Venkatesh, G., et al. (2017) · arXiv · doi:10.48550/arxiv.1710.03740 · Cited by 1,179 · cited 16× · see citing works
“The computations during forward pass and back propagation use FP16 datatype while results are accumulated into FP32.”
BackgroundMethodInfluentialDeep Learning Recommendation Model for Personalization and Recommendation Systems
Naumov, M., Mudigere, D., Shi, H.-J. M., Huang, J., Sundaraman, N., Park, J., Wang, X., Gupta, U., Wu, C.-J., Azzolini, A. G., et al. (2019) · arXiv · doi:10.48550/arxiv.1906.00091 · Cited by 411 · cited 2× · see citing works
“The accuracy of the recommendation system models is measured by the log loss [38, 43, 29], which predicts when the users will click on ads.”
MethodSelf-citationTraining deep neural networks with low precision multiplications
Courbariaux, M., Bengio, Y., & David, J.-P. (2014) · arXiv · doi:10.48550/arxiv.1412.7024 · Cited by 407 · see citing works
Flexpoint: An Adaptive Numerical Format for Efficient Training of Deep Neural NetworksPreprint
Köster, U., Webb, T. J., Wang, X., Nassar, M., Bansal, A. K., Constable, W., Elibol, O. H., Gray, S., Hall, S., Hornof, L., et al. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1711.02213 · Cited by 236 · cited 4× · see citing works
“Literature shows that various different reduced precision data representations have been investigated which can be broadly classified into two types: the more standard floating-point based formats [28, 15, 12] and custom fixed point based formats [37, 7, 18, 22, 23].”
BackgroundInfluential8-Bit Approximations for Parallelism in Deep LearningPreprint
Dettmers, T. (2015) · International Conference on Learning Representations · doi:10.48550/arxiv.1511.04561 · Cited by 205 · cited 1× · see citing works
“Literature shows that various different reduced precision data representations have been investigated which can be broadly classified into two types: the more standard floating-point based formats [26, 13, 10] and custom fixed point based formats [34, 6, 16, 20, 21].”
BackgroundMixed Precision Training of Convolutional Neural Networks using Integer OperationsPreprint
Das, D., Mellempudi, N., Mudigere, D., Kalamkar, D. D., Avancha, S., Banerjee, K., Sridharan, S., Vaidyanathan, K., Kaul, B., Georganas, E., et al. (2018) · International Conference on Learning Representations · doi:10.48550/arxiv.1802.00930 · Cited by 177 · cited 3× · see citing works
“There are at least three half-precision formats in the domain of mixed precision training of large neural networks: FP16 [26], 16-bit Integer based [8], and BFLOAT16.”
BackgroundSelf-citationImage Classification at Supercomputer ScalePreprint
Ying, C., Kumar, S., Chen, D., Wang, T., & Cheng, Y.-L. (2018) · arXiv.org · doi:10.48550/arxiv.1811.06992 · Cited by 133 · see citing works
Anatomy of High-Performance Deep Learning Convolutions on SIMD Architectures
Georganas, E., Avancha, S., Banerjee, K., Kalamkar, D. D., Henry, G., Pabst, H., & Heinecke, A. (2018) · International Conference for High Performance Computing, Networking, Storage and Analysis · doi:10.48550/arxiv.1808.05567 · Cited by 118
Self-citationTernary Neural Networks with Fine-Grained QuantizationPreprint
Mellempudi, N., Kundu, A., Mudigere, D., Das, D., Kaul, B., & Dubey, P. K. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1705.01462 · Cited by 89 · cited 1× · see citing works
“Inference has witnessed a proliferation of mixed precision compute [19, 33, 27, 25] where different operations execute at different precision, all the way from binary/ternary operands to 16b floating point.”
BackgroundSelf-citationHigh Performance Monte Carlo Simulation of Ising Model on TPU Clusters
Yang, K., Chen, Y.-F., Roumpos, G., Colby, C., & Anderson, J. R. (2019) · International Conference for High Performance Computing, Networking, Storage and Analysis · doi:10.1145/3295500.3356149 · Cited by 57 · see citing works
Dynamically scaled fixed point arithmetic
Williamson, D. (1991) · IEEE Pacific Rim Conference on Communications, Computers and Signal Processing. Conference Proceedings · doi:10.1109/pacrim.1991.160742 · Cited by 47 · cited 1×
“The work reported in [34] leverages the dynamically scaled fixed point representation proposed in [36] to speed up convolution neural networks by 4× over an optimized floating point implementation on general purpose CPU hardware.”
BackgroundAutomatic Full Compilation of Julia Programs and ML Models to Cloud TPUs
Fischer, K., & Saba, E. (2018) · arXiv.org · doi:10.48550/arxiv.1810.09868 · Cited by 20
Ternary Residual NetworksPreprint
Kundu, A., Banerjee, K., Mellempudi, N., Mudigere, D., Das, D., Kaul, B., & Dubey, P. (2017) · arXiv.org · doi:10.48550/arxiv.1707.04679 · Cited by 9 · cited 1× · see citing works
“Inference has witnessed a proliferation of mixed precision compute [19, 33, 27, 25] where different operations execute at different precision, all the way from binary/ternary operands to 16b floating point.”
BackgroundSelf-citation
Flagged first, then by citations
9 of the 43 references couldn’t be matched to a known paper.
Research works printed but not linked: 5
- [10][10] Jeffrey Dean, Greg Corrado, Rajat Monga, Kai Chen, Matthieu Devin, Mark Mao, Andrew Senior, Paul Tucker, Ke Yang, Quoc V Le, et al. Large scale distributed deep networks. In Advances in neural information processing systems , pages 1223-1231, 2012. -
- [15][15] Boris Ginsburg, Sergei Nikolaev, and Paulius Micikevicius. Training of deep networks with half-precision float. NVidia GPU Technology Conference , 2017. -
- [34][34] Matthew Richardson, Ewa Dominowska, and Robert Ragno. Predicting clicks: Estimating the click-through rate for new ads. In Proceedings of the 16th International Conference on World Wide Web , WWW'07, pages 521-530, New York, NY, USA, 2007. ACM. -
- [37][37] Vincent Vanhoucke, Andrew Senior, and Mark Z Mao. Improving the speed of neural networks on CPUs. In Proc. Deep Learning and Unsupervised Feature Learning NIPS Workshop , volume 1, 2011. -
- [43][43] Jian Zhang, Jiyan Yang, and Hector Yuen. Training with low-precision embedding tables. In Systems for Machine Learning Workshop at NeurIPS 2018 . 2018.
Cited by
Sorted by citations
Large Language Models in Finance: A Survey
Li, Y., Wang, S., Ding, H., & Chen, H. (2023) · ACM International Conference on AI in Finance (ICAIF) · doi:10.1145/3604237.3626869 · Cited by 334 · see citing works
“In Section 5, we provide a more detailed discussion of which option is more favorable under different circumstances.”
BackgroundTen Lessons From Three Generations Shaped Google’s TPUv4i : Industrial Product
Jouppi, N., Yoon, D. H., Ashcraft, M. B., Gottscho, M., Jablin, T., Kurian, G., Laudon, J., Li, S., Ma, P. C., Ma, X., et al. (2021) · International Symposium on Computer Architecture · doi:10.1109/isca52012.2021.00010 · Cited by 483 · see citing works
“Bfloat16 is a better match to DNNs than IEEE 754 fp16 [24], so the MXU was changed to become the first hardware to support it, with 16K MAC units (½ the side of the systolic array, so ¼ of the size).”
FP8 Formats for Deep LearningPreprint
Micikevicius, P., Stosic, D., Burgess, N., Cornea, M., Dubey, P., Grisenthwaite, R., Ha, S., Heinecke, A., Judd, P., Kamalu, J., et al. (2022) · arXiv.org · doi:10.48550/arxiv.2209.05433 · Cited by 352 · see citing works
“Common floating point types for training include IEEE single precision, TF32 mode for single precision [19], IEEE half precision [14], and bfloat16 [9].”
Self-citationA domain-specific supercomputer for training deep neural networks
Jouppi, N., Yoon, D., Kurian, G., Li, S., Patil, N., Laudon, J., Young, C., & Patterson, D. A. (2020) · Communications of the ACM · doi:10.1145/3360307 · Cited by 312 · see citing works
“Beyond our experience that it works for training production applications, a recent Intel study corroborated its benefits.(21) However, fp16 requires adjustments to training software (loss scaling) to deliver convergence and efficiency.”
MethodResultEfficient Processing of Deep Neural Networks
Sze, V., Chen, Y., Yang, T.-J., & Emer, J. (2020) · Synthesis lectures on computer architecture · doi:10.2200/s01004ed1v01y202004cac050 · Cited by 117 · see citing works
NVILA: Efficient Frontier Visual Language Models
Liu, Z., Zhu, L., Shi, B., Zhang, Z., Lou, Y., Yang, S. W., Xi, H., Cao, S., Gu, Y., Li, D., et al. (2025) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52734.2025.00390 · Cited by 73 · see citing works
“FP16 [31] and BF16 [32] are standard precisions for model training, since they offer acceleration without accuracy loss, supported natively by NVIDIA GPUs.”
The Deep Learning Compiler: A Comprehensive Survey
Li, M., Liu, Y., Liu, X., Sun, Q., You, X., Yang, H.-L., Luan, Z.-Z., Gan, L., Yang, G.-W., & Qian, D. (2020) · IEEE Transactions on Parallel and Distributed Systems · doi:10.1109/tpds.2020.3030548 · Cited by 261 · see citing works
“Besides, low-precision training and inference, including FP32, FP16, INT8, and non-IEEE floating-point format bfloat16 [45] are also supported.”
Degree-Quant: Quantization-Aware Training for Graph Neural NetworksPreprint
Tailor, S. A., Fernández-Marqués, J., & Lane, N. D. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2008.05000 · Cited by 136 · see citing works
“, 2017) and data formats (Carmichael et al., 2018; Kalamkar et al., 2019), it is 8-bit integer (INT8) quantization that has attracted the most attention.”
BackgroundEfficient Methods for Natural Language Processing: A Survey
Treviso, M., Lee, J.-U., Ji, T., van Aken, B., Cao, Q., Ciosici, M. R., Hassid, M., Heafield, K., Hooker, S., Raffel, C., et al. (2023) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00577 · Cited by 124 · see citing works
Reducing Transformer Key-Value Cache Size with Cross-Layer Attention
Brandon, W., Mishra, M., Nrusimha, A., Panda, R., & Ragan-Kelley, J. (2024) · Neural Information Processing Systems · doi:10.48550/arxiv.2405.12981 · Cited by 172
“We use mixed precision training [Micikevicius et al., 2018] in BF16 [Kalamkar et al., 2019] with gradient all-reduce and gradient accumulation in FP32 for training stability.”
MethodGranite Code Models: A Family of Open Foundation Models for Code IntelligencePreprint
Mishra, M., Stallone, M., Zhang, G., Shen, Y., Prasad, A., Soria, A. M., Merler, M., Selvam, P., Surendran, S., Singh, S., et al. (2024) · arXiv.org · doi:10.48550/arxiv.2405.04324 · Cited by 172 · see citing works
“We use Megatron’s distributed optimizer with mixed precision training (Micikevicius et al., 2018) in BF16 (Kalamkar et al., 2019) with gradient all-reduce and gradient accumulation in FP32 for training stability.”
MethodA Survey of Techniques for Optimizing Transformer Inference
Chitty-Venkata, K. T., Mittal, S., Emani, M., Vishwanath, V., & Somani, A. (2023) · Journal of systems architecture · doi:10.48550/arxiv.2307.07982 · Cited by 169
“formats such as Bfloat16 [158], Microsoft Floating Point (MSFP) [159], Tensorfloat32 [160], and FP8 [161, 162] provide better computational efficiency and accuracy for several transformer workloads.”
BackgroundInfluentialA Survey of Techniques for Optimizing Transformer Inference
Chitty-Venkata, K. T., Mittal, S., Emani, M., Vishwanath, V., & Somani, A. K. (2023) · Journal of Systems Architecture · doi:10.1016/j.sysarc.2023.102990 · Cited by 154 · see citing works
An Overview of Neural Network CompressionPreprint
O’Neill, J. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.03669 · Cited by 93 · see citing works
KERPLE: Kernelized Relative Positional Embedding for Length ExtrapolationPreprint
Chi, T.-C., Fan, T.-H., Ramadge, P., & Rudnicky, A. I. (2022) · Neural Information Processing Systems · doi:10.48550/arxiv.2205.09921 · Cited by 119 · see citing works
DART-Math: Difficulty-Aware Rejection Tuning for Mathematical Problem-Solving
Tong, Y., Zhang, X., Wang, R., Wu, R., & He, J. (2024) · Neural Information Processing Systems · doi:10.48550/arxiv.2407.13690 · Cited by 118
“\n\n###Instruction:\n{query}\n\n### Response:\n Other Details: For efficiency, We utilize various tools / libraries / techniques including: • the DeepSpeed distributed framework (Rasley et al., 2020) with ZeRO (Rajbhandari et al., 2020) stage 3 • gradient checkpointing (Chen et al., 2016) • torch.compile (Ansel et al.…”
LN3Diff++: Scalable Latent Neural Fields Diffusion for Speedy 3D Generation
Lan, Y., Hong, F., Zhou, S.-C., Yang, S., Meng, X., Chen, Y.-W., Lyu, Z., Dai, B., Pan, X.-G., & Loy, C. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2025.3633073 · Cited by 118 · see citing works
“For efficient and scalable training, we leverage BFloat16 [52] with FlashAttention [18], [19] enabled.”
Filtering, Distillation, and Hard Negatives for Vision-Language Pre-Training
Radenovic, F., Dubey, A., Kadian, A., Mihaylov, T., Vandenhende, S., Patel, Y., Wen, Y., Ramanathan, V., & Mahajan, D. (2023) · Computer Vision and Pattern Recognition · doi:10.1109/cvpr52729.2023.00673 · Cited by 117 · see citing works
Dave, S., Baghdadi, R., Nowatzki, T., Avancha, S., Shrivastava, A., & Li, B. (2021) · Proceedings of the IEEE · doi:10.1109/jproc.2021.3098483 · Cited by 110 · see citing works
Self-citationWhy Do We Need Weight Decay in Modern Deep Learning?
D'Angelo, F., Andriushchenko, M., Varre, A., & Flammarion, N. (2023) · Neural Information Processing Systems · doi:10.48550/arxiv.2310.04415 · Cited by 107
“Using bfloat16 training helps to significantly speed up training and reduce the GPU memory requirements allowing to train larger models and use larger batches (Kalamkar et al., 2019).”
BackgroundA TinyML Platform for On-Device Continual Learning With Quantized Latent Replays
Ravaglia, L., Rusci, M., Nadalini, D., Capotondi, A., Conti, F., & Benini, L. (2021) · IEEE Journal on Emerging and Selected Topics in Circuits and Systems · doi:10.1109/jetcas.2021.3121554 · Cited by 101 · see citing works
“The cluster includes a set of four highly flexible FPUs shared between all nine cores, capable of FP32 and FP16 computation [57].”
MethodPredicting cellular responses to perturbation across diverse contexts with State
Adduri, A. K., Gautam, D., Bevilacqua, B., Naghipourfar, M., Imran, A., Shah, R., Teyssier, N., Verma, R., Carpenter, C., Eraslan, B., et al. (2026) · Cell · doi:10.1016/j.cell.2026.07.052 · Cited by 100 · see citing works
With Shared Microexponents, A Little Shifting Goes a Long Way
Rouhani, B. D., Zhao, R., Elango, V., Shafipour, R., Hall, M., Mesmakhosroshahi, M., More, A., Melnick, L., Golub, M., Varatkar, G., et al. (2023) · International Symposium on Computer Architecture · doi:10.1145/3579371.3589351 · Cited by 98 · see citing works
Self-citationSecuring Approximate Homomorphic Encryption Using Differential Privacy
Li, B.-Y., Micciancio, D., Schultz, M., & Sorrell, J. (2022) · IACR Cryptology ePrint Archive · doi:10.1007/978-3-031-15802-5_20 · Cited by 96
“Some applications may solely require 8 or 16 bits of precision (see for example [16] or [33] for work on training ML models with low-precision computations).”
PANTHER: A Programmable Architecture for Neural Network Training Harnessing Energy-Efficient ReRAM
Ankit, A., Hajj, I. E., Chalamalasetti, S. R., Agarwal, S., Marinella, M., Foltin, M., Strachan, J. P., Milojicic, D., Hwu, W., & Roy, K. (2019) · IEEE transactions on computers · doi:10.1109/tc.2020.2998456 · Cited by 87 · see citing works
Lu, D., Wu, H., Liang, J.-Q., Xu, Y., He, Q., Geng, Y.-P., Han, M., Xin, Y., & Xiao, Y. (2023) · arXiv.org · doi:10.48550/arxiv.2302.09432 · Cited by 86
“In particular, we found that using the BFLOAT16 (Kalamkar et al., 2019) half-precision floating-point format for optimization can effectively solve the problem of gradient overflow that occurs in the training process with FP16 halfprecision floating-point format, without the need to repeatedly…”
BackgroundMethodInfluentialTowards Cross-Tokenizer Distillation: the Universal Logit Distillation Loss for LLMs
Boizard, N., Haddad, K. E., Hudelot, C., & Colombo, P. (2024) · Trans. Mach. Learn. Res. · doi:10.48550/arxiv.2402.12030 · Cited by 82
Defeating the Training-Inference Mismatch via FP16
Qi, P., Liu, Z.-C., Zhou, X.-X., Pang, T.-Y., Du, C., Lee, W.-S., & Lin, M. (2025) · arXiv.org · doi:10.48550/arxiv.2510.26788 · Cited by 78
“We find that merely switching the training precision from the now-dominant BF16 format [Dean et al., 2012, Kalamkar et al., 2019] to the earlier Float16 (FP16) format [Micikevicius et al., 2017] substantially mitigates the policy mismatch and yields significant performance improvements across RL…”
nuQmm: Quantized MatMul for Efficient Inference of Large-Scale Generative Language ModelsPreprint
Park, G., Park, B., Kwon, S., Kim, B., Lee, Y., & Lee, D. (2022) · arXiv.org · doi:10.48550/arxiv.2206.09557 · Cited by 77 · see citing works
“As an example, BFLOAT16 [47] has 16 bits to represent a parameter, and thus, saves memory footprint and area overhead of arithmetic units compared to FP32.”
ResultKwon, D.-H., Lee, S., Kim, K., Oh, S., Park, J., Hong, G., Ka, D., Hwang, K.-D., Park, J.-J., Kang, K., et al. (2023) · IEEE Journal of Solid-State Circuits · doi:10.1109/jssc.2022.3200718 · Cited by 75
“The weight and the vector are bfloat16 (Brain Floating Point, BF16) format; it consists of 1-b sign (S), 8-b exponent (EX), and 7-b mantissa (MA) [6].”
BackgroundSelf-citationImproving Neural Network Efficiency via Post-training Quantization with Adaptive Floating-Point
Liu, F., Zhao, W., He, Z., Wang, Y., Wang, Z., Dai, C., Liang, X., & Jiang, L. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00523 · Cited by 67 · see citing works
“Specifically, FP32 format has an 8-bit exponent and 23-bit mantissa, FP16 has a 5-bit exponent and 10-bit mantissa, BF16 [16] has an 8-bit exponent and 7-bit mantissa, and TF32 has an 8-bit exponent and 10-bit mantissa.”
BackgroundMethodEvaluations on Deep Neural Networks Training Using Posit Number System
Lu, J.-M., Fang, C., Xu, M., Lin, J., & Wang, Z.-F. (2020) · IEEE transactions on computers · doi:10.1109/tc.2020.2985971 · Cited by 75 · see citing works
“Besides, the modified FP16 (BFLOAT16) format [16] was studied for DNN training.”
MethodEfficient training of large language models on distributed infrastructures: a survey
Duan, J., Zhang, S., Wang, Z., Jiang, L.-J., Qu, W., Hu, Q.-H., Wang, G., Weng, Q.-Z., Yan, H., Zhang, X.-C., et al. (2024) · Vicinagearth · doi:10.1007/s44336-026-00038-z · Cited by 71
“By employing lazy softmax, the division by j e s j in softmax can be delayed to the very end of the attention operation.”
Jaques, S., & Rattew, A. G. (2025) · Quantum · doi:10.22331/q-2025-12-02-1922 · Cited by 45 · see citing works
TensorDash: Exploiting Sparsity to Accelerate Deep Neural Network Training
Mahmoud, M., Edo, I., Zadeh, A. H., Awad, O. M., Pekhimenko, G. G., Albericio, J., & Moshovos, A. (2020) · IEEE/ACM International Symposium on Microarchitecture (MICRO) · doi:10.1109/micro50266.2020.00069 · Cited by 76 · see citing works
“bfloat16 [26], [41], [73], dynamic floating-point [14], and flexpoint [42]).”
MethodFitzGerald, J., Ananthakrishnan, S., Arkoudas, K., Bernardi, D., Bhagia, A., Delli Bovi, C., Cao, J., Chada, R., Chauhan, A., Chen, L., et al. (2022) · Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining · doi:10.1145/3534678.3539173 · Cited by 68 · see citing works
“Another way to mitigate the stability issues is to use BFLOAT16 [18], which was not available in Deepspeed at the time of our experiments.”
MethodFine Tuning LLM for Enterprise: Practical Guidelines and Recommendations
MathavRaj, J., Kushala, V., Warrier, H., & Gupta, Y. (2024) · arXiv.org · doi:10.48550/arxiv.2404.10779 · Cited by 66
“Brain float 16[15] or BF16 maintains the same range as FP32 but reduction of precision to 7 bits.”
BackgroundExploring the Role of Large Language Models in Prompt Encoding for Diffusion Models
Ma, B., Zong, Z.-F., Song, G.-L., Li, H.-S., & Liu, Y. (2024) · Neural Information Processing Systems · doi:10.48550/arxiv.2406.11831 · Cited by 61
“Besides, considering the broader numerical representation range of bfloat16, we finally use the bfloat16 mixed precision training [24] strategy.”
Michaud, E. J., Liu, Z.-M., & Tegmark, M. (2022) · Entropy · doi:10.3390/e25010175 · Cited by 58 · see citing works
“Accordingly, to save memory and speed up computation, much work has gone into reducing the numerical precision used in models without sacrificing model performance much [1, 2, 3].”
Intriguing Properties of Quantization at ScalePreprint
Ahmadian, A., Dash, S., Chen, H.-Y., Venkitesh, B., Gou, S., Blunsom, P., Ustun, A., & Hooker, S. (2023) · Neural Information Processing Systems · doi:10.48550/arxiv.2305.19268 · Cited by 56 · see citing works
“In this technique, a copy of weights is stored in full-precision ( fp32 ) whereas the forward and backward passes are done using half-precision in either float16 ( fp16 ) or bfloat16 ( bf16 ) (Kalamkar et al., 2019; Dean et al., 2012; Abadi et al., 2015).”
MethodDefensive approximation: securing CNNs using approximate computing
Guesmi, A., Alouani, I., Khasawneh, K. N., Baklouti, M., Frikha, T., Abid, M., & Abu-Ghazaleh, N. B. (2020) · International Conference on Architectural Support for Programming Languages and Operating Systems · doi:10.1145/3445814.3446747 · Cited by 55 · see citing works
“The Bfloat16 (Brain Floating Point) [28] is a truncated version of the 32-bit IEEE 754 single-precision floating-point format (float32).”
BackgroundMaji, S., Banerjee, U., & Chandrakasan, A. P. (2021) · IEEE Internet of Things Journal · doi:10.1109/jiot.2021.3061314 · Cited by 56 · see citing works
Optimizing Deep Learning Recommender Systems Training on CPU Cluster Architectures
Kalamkar, D. D., Georganas, E., Srinivasan, S., Chen, J., Shiryaev, M., & Heinecke, A. (2020) · International Conference for High Performance Computing, Networking, Storage and Analysis · doi:10.1109/sc41405.2020.00047 · Cited by 52 · see citing works
“Classic training approaches using BFLOAT16, FP16 or even int16 require so-called master-weights, a full precision FP32 copy of the entire model throughout the full training process [34]–[36].”
Self-citationIntel Nervana Neural Network Processor-T (NNP-T) Fused Floating Point Many-Term Dot Product
Hickmann, B. J., Chen, J., Rotzin, M., Yang, A., Urbanski, M., & Avancha, S. (2020) · IEEE Symposium on Computer Arithmetic · doi:10.1109/arith48897.2020.00029 · Cited by 51
“…properties, The first major design decision was to select BFloat16 (BF16) as the numerical format over several alternatives such as FP32, FP16, and various integer formats, as it offered the power and area efficiency of a 16-bit format with convergence characteristics similar to that of FP32 [6].”
MethodSelf-citationA TensorFlow Simulation Framework for Scientific Computing of Fluid Flows on Tensor Processing Units
Wang, Q., Ihme, M., Chen, Y.-F., & Anderson, J. R. (2021) · Computer Physics Communications · doi:10.1016/j.cpc.2022.108292 · Cited by 50
“Software emulation with multiple passes of the MXU operations makes higher-precision matrix multiplication (32-bit and 64-bit) possible, with a performance trade-off [15].”
Lee, J., Mukhanov, L., Molahosseini, A. S., Minhas, U. I., Yang, H., del Rincón, J. M., Dichev, K., Hong, C.-H., & Vandierendonck, H. (2023) · ACM Computing Surveys · doi:10.1145/3587095 · Cited by 55 · see citing works
“A conventional mixed-precision training applied lowerprecision arithmetic to the multiplications in MACs, including both forward and backward paths, and higher-precision arithmetic to the accumulations in the MACs using lower-precision quantized operands [95, 125].”
MethodFastFold: Reducing AlphaFold Training Time from 11 Days to 67 HoursPreprint
Cheng, S., Zhao, X.-L., Lu, G.-Y., Fang, J.-R., Yu, Z.-M., Zheng, T.-Y., Wu, R., Zhang, X., Peng, J., & You, Y. (2022) · arXiv.org · doi:10.48550/arxiv.2203.00854 · Cited by 48 · see citing works
“To speed up training time and reduce memory consumption, AlphaFold training uses Bfloat16 precision [9].”
MethodReasoning-to-Defend: Safety-Aware Reasoning Can Defend Large Language Models from Jailbreaking
Zhu, J., Yan, L., Wang, S.-Q., Yin, D.-W., & Sha, L. (2025) · Conference on Empirical Methods in Natural Language Processing · doi:10.48550/arxiv.2502.12970 · Cited by 47
“We conduct Distributed Data Parallel (DDP, Li et al., 2020) with mixed-precision training with bfloat16 data type (Kalamkar et al., 2019) implemented in apex 4 .”
COAT: Compressing Optimizer states and Activation for Memory-Efficient FP8 Training
Xi, H.-C., Cai, H., Zhu, L.-G., Lu, Y., Keutzer, K., Chen, J.-F., & Han, S. (2024) · International Conference on Learning Representations · doi:10.48550/arxiv.2410.19313 · Cited by 46
“BF16 training (Kalamkar et al., 2019) refines this approach, as BF16 has a larger representation range and is more stable for large-scale training.”
BackgroundMethodRethinking Data Selection for Supervised Fine-TuningPreprint
Shen, M. (2024) · arXiv.org · doi:10.48550/arxiv.2402.06094 · Cited by 44 · see citing works
“All models are trained with BFloat16 (Kalamkar et al., 2019) mixed precision for stability.”
MethodE-prop on SpiNNaker 2: Exploring online learning in spiking RNNs on neuromorphic hardware
Rostami, A., Vogginger, B., Yan, Y., & Mayr, C. (2022) · Frontiers in Neuroscience · doi:10.3389/fnins.2022.1018006 · Cited by 44 · see citing works
“, 2022) or mixedprecision training (lower precision data format for some variables) (Micikevicius et al., 2018; Kalamkar et al., 2019), lowrank matrix representation (Kusupati et al.”
BackgroundMixed-Precision Embedding Using a CachePreprint
Yang, J., Huang, J., Park, J., Tang, P. T. P., & Tulloch, A. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.11305 · Cited by 27 · see citing works
“Notable works using this approach include (Gupta et al., 2015; Chen et al., 2017; De Sa et al., 2018; Wu et al., 2018b; Zhang et al., 2018; Kalamkar et al., 2019).”
Self-citationSENECA: building a fully digital neuromorphic processor, design trade-offs and challenges
Tang, G., Vadivel, K., Xu, Y., Bilgic, R., Shidqi, K., Detterer, P., Traferro, S., Konijnenburg, M., Sifalakis, M., van Schaik, G.-J., et al. (2023) · Frontiers in Neuroscience · doi:10.3389/fnins.2023.1187252 · Cited by 26 · see citing works
“They also support shared scale factors (Köster et al., 2017; Moons et al., 2017; Jacob et al., 2018; Kalamkar et al., 2019; Coelho et al., 2021).”
BackgroundSAVE: Sparsity-Aware Vector Engine for Accelerating DNN Training and Inference on CPUs
Gong, Z., Ji, H., Fletcher, C., Hughes, C., Baghsorkhi, S. S., & Torrellas, J. (2020) · Micro · doi:10.1109/micro50266.2020.00070 · Cited by 43 · see citing works
“Training in BF16 yields an accuracy comparable to that of using FP32, without tuning hyperparameters [39], [65].”
BackgroundSearching for Winograd-aware Quantized NetworksPreprint
Fernández-Marqués, J., Whatmough, P. N., Mundy, A., & Mattina, M. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2002.10711 · Cited by 28 · see citing works
Four Over Six: More Accurate NVFP4 Quantization with Adaptive Block Scaling
Cook, J., Guo, J.-X., Xiao, G.-X., Lin, Y.-J., Wyss, K., Nazemi, M., Mishra, A., del Mundo, C. C., Blankevoort, T., & Han, S. (2025) · arXiv.org · doi:10.48550/arxiv.2512.02010 · Cited by 42
“An example of this can be seen in numerical precision: it used to be standard to train models using FP32, then FP16 [1], and now it is standard to keep most or all parameters in BF16 [2].”
GNNear: Accelerating Full-Batch Training of Graph Neural Networks with near-Memory Processing
Zhou, Z., Li, C., Wei, X., Wang, X., & Sun, G. (2021) · International Conference on Parallel Architectures and Compilation Techniques · doi:10.1145/3559009.3569670 · Cited by 41 · see citing works
“BF16 has the same accuracy as FP32 for NN training [47] but is more cost-efficient.”
MethodEvaluation Method of Deep Learning-Based Embedded Systems for Traffic Sign Detection
Lopez-Montiel, M. A., Orozco-Rosas, U., Sánchez-Adame, M., Picos, K., & Montiel, O. (2021) · IEEE Access · doi:10.1109/access.2021.3097969 · Cited by 55 · see citing works
“A regular TPU Cloud Service has two 128 × 128 systolic arrays, adding 32,768 ALUs for 16-bit floating-point values on a single processor [72].”
BackgroundZhang, T., Hariri, M., Sui, Y., Zhong, S.-C., Chaudhary, V., Hu, X. B., & Shrivastava, A. (2025) · Neural Information Processing Systems · doi:10.48550/arxiv.2504.11651 · Cited by 40
“Compared to FP16, BF16 offers a wider dynamic range at the cost of precision, which improves numerical stability and mitigates overflow issues during training [8, 19].”
Vision-LSTM: xLSTM as Generic Vision Backbone
Alkin, B., Beck, M., Pöppel, K., Hochreiter, S., & Brandstetter, J. (2024) · arXiv · doi:10.48550/arxiv.2406.04303 · Cited by 39
Multi-lingual Evaluation of Code Generation Models
Athiwaratkun, B., Gouda, S. K., Wang, Z., Li, X., Tian, Y., Tan, M., Ahmad, W. U., Wang, S., Sun, Q., Shang, M., et al. (2022) · arXiv · doi:10.48550/arxiv.2210.14868 · Cited by 38 · see citing works
“Our training pipeline is based on PyTorch Lightning § and we use bfloat16 (Kalamkar et al., 2019) and DeepSpeed (Rasley et al.”
MethodLearning compositional functions via multiplicative weight updatesPreprint
Bernstein, J., Zhao, J., Meister, M., Liu, M.-Y., Anandkumar, A., & Yue, Y. (2020) · Neural Information Processing Systems · doi:10.48550/arxiv.2006.14560 · Cited by 38 · see citing works
Impact of Mixed Precision Techniques on Training and Inference Efficiency of Deep Neural Networks
Dörrich, M., Fan, M.-C., & Kist, A. (2023) · IEEE Access · doi:10.1109/access.2023.3284388 · Cited by 38 · see citing works
“Moreover, it was shown that bfloat16 training of neural networks can achieve the same state-of-the-art results as the standard 32-bit format [10].”
BackgroundVarious Lengths, Constant Speed: Efficient Language Modeling with Lightning AttentionPreprint
Qin, Z., Sun, W.-G., Li, D., Shen, X.-Y., Sun, W., & Zhong, Y.-R. (2024) · International Conference on Machine Learning · doi:10.48550/arxiv.2405.17381 · Cited by 37 · see citing works
Meteor: Mamba-based Traversal of Rationale for Large Language and Vision Models
Lee, B.-K., Kim, C. W., Park, B., & Yong, M.-R. (2024) · Neural Information Processing Systems · doi:10.48550/arxiv.2405.15574 · Cited by 37
Training and inference of large language models using 8-bit floating point
Perez, S. P., Zhang, Y., Briggs, J., Blake, C., Levy-Kramer, J., Balanca, P., Luschi, C., Barlow, S., & Fitzgibbon, A. W. (2023) · arXiv.org · doi:10.48550/arxiv.2309.17224 · Cited by 37
“Models of more than 100 billion parameters like Bloom [Scao et al., 2022] or OPT [Zhang et al., 2022] struggled to find a stable loss scaling for FP16 and ended up employing BF16.”
MethodInfluentialDaghaghi, S., Meisburger, N., Zhao, M., Wu, Y., Gobriel, S., Tai, C., & Shrivastava, A. (2021) · Conference on Machine Learning and Systems · doi:10.48550/arxiv.2103.10891 · Cited by 37
“Brain floating-point format (BF16) uses 16 bits in computer memory to represent a floating-point number (Kalamkar et al., 2019).”
BackgroundUnlocking the Power of Inline Floating-Point Operations on Programmable Switches
Yuan, Y.-F., Alama, O., Sapio, A., Fei, J., Nelson, J., Ports, D. R. K., Canini, M., & Kim, N. S. (2021) · Symposium on Networked Systems Design and Implementation · doi:10.48550/arxiv.2112.06095 · Cited by 36
Revisiting BFloat16 TrainingPreprint
Zamirai, P., Zhang, J., Aberger, C. R., & De, C. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.06192 · Cited by 15 · see citing works
“State-of-the-art low-precision generic training algorithms use mixed precision, still requiring accelerators to contain 32-bit compute units [7, 8, 42].”
BackgroundMethodATM: Adversarial Tuning Multi-agent System Makes a Robust Retrieval-Augmented Generator
Zhu, J., Yan, L., Shi, H.-B., Yin, D.-W., & Sha, L. (2024) · Conference on Empirical Methods in Natural Language Processing · doi:10.48550/arxiv.2405.18111 · Cited by 35
“To this end, we use mixed-precision training with a model in bfloat16 data type (Kalamkar et al., 2019) implemented in apex and ZeRO (Rajbhandari et al., 2020) stage 1 implemented with DeepSpeed 8 which splits optimizer state across GPUs thus saving memory.”
MethodMixed-precision numerics in scientific applications: survey and perspectives
Kashi, A., Lu, H., Brewer, W., Rogers, D., Matheson, M. A., Shankar, M., & Wang, F.-Y. (2024) · Journal of Supercomputing · doi:10.48550/arxiv.2412.19322 · Cited by 34
“Brain Float 16, usually referred to as BFloat16 or BF16, was developed by Google specifically for deep learning training on their Tensor Processing Units (TPUs) [9], and has since become a widely adopted standard.”
First-Generation Inference Accelerator Deployment at FacebookPreprint
Anderson, M. J., Chen, B., Chen, S., Deng, S., Fix, J., Gschwind, M., Kalaiah, A., Kim, C., Lee, J., Liang, J., et al. (2021) · arXiv.org · doi:10.48550/arxiv.2107.04140 · Cited by 33 · see citing works
“Since recent training hardware supports bfloat16 [29], [31], [43], bfloat16 support in inference accelerators can be useful to unify numerical format between”
BackgroundSelf-citationField-Programmable Gate Array Architecture for Deep Learning: Survey and Future Directions
Boutros, A., Arora, A., & Betz, V. (2024) · Proceedings of the IEEE · doi:10.1109/jproc.2025.3623023 · Cited by 32 · see citing works
“Each PE in the tensor block can implement 1 × int16 , fp16 , or 16-bit brain floating-point ( bfloat16 ) [129] MAC, and it can also be fractured to implement 4 × int8 MACs.”
An approach to generate correctly rounded math libraries for new floating point variants
Lim, J. P., Aanjaneya, M., Gustafson, J. L., & Nagarakatte, S. (2021) · Proc. ACM Program. Lang. · doi:10.1145/3434310 · Cited by 32
“Bfloat16 ’s dynamic range is similar to a 32-bit float but has lower memory traffic and footprint, which makes it appealing for neural networks [Kalamkar et al. 2019].”
BackgroundQA-LIGN: Aligning LLMs through Constitutionally Decomposed QA
Dineen, J., Rrv, A., Liu, Q., Xu, Z., Ye, X., Shen, M., Li, Z.-N., Lu, S.-J., Baral, C., Chen, M., et al. (2025) · Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2025.findings-emnlp.1123 · Cited by 31
“All models were fine-tuned with a maximum sequence length of 1,536 tokens and run in BFloat16 mixed precision (Kalamkar et al., 2019)..”
Pre-training Distillation for Large Language Models: A Design Space Exploration
Peng, H., Lv, X., Bai, Y., Yao, Z.-J., Zhang, J.-J., Hou, L., & Li, J.-Z. (2024) · Annual Meeting of the Association for Computational Linguistics · doi:10.48550/arxiv.2410.16215 · Cited by 31
BioTrove: A Large Curated Image Dataset Enabling AI for Biodiversity
Yang, C.-H., Feuer, B., Jubery, T., Deng, Z. K., Nakkab, A., Hasan, M. Z. Z., Chiranjeevi, S., Marshall, K. O., Baishnab, N., Singh, A., et al. (2024) · Neural Information Processing Systems · doi:10.52202/079017-3241 · Cited by 31
“We train our models for a combined 10 days on 8xH100 nodes in bfloat16 precision [18] with gradient checkpointing, computing loss with local features, and utilizing static graph optimization for DDP.”
A Reconfigurable Processing Element for Multiple-Precision Floating/Fixed-Point HPC
Li, B.-Y., Li, K., Zhou, J.-J., Ren, Y., Mao, W., Yu, H., & Wong, N. (2024) · IEEE Transactions on Circuits and Systems - II - Express Briefs · doi:10.1109/tcsii.2023.3322259 · Cited by 31
“These parts also appear in BF16 [18] and TF32 [19], but different floating-point numbers have different exponent and mantissa bitwidths as well as exponent biases ( bias ) .”
BackgroundHarnessing Deep Learning via a Single Building Block
Georganas, E., Banerjee, K., Kalamkar, D. D., Avancha, S., Venkat, A., Anderson, M. R., Henry, G., Pabst, H., & Heinecke, A. (2020) · IEEE International Parallel and Distributed Processing Symposium (IPDPS) · doi:10.1109/ipdps47924.2020.00032 · Cited by 40 · see citing works
“The results in section IV are in single precision (FP32), however we already have implementations supporting the int8 and bfloat16 datatypes (via the new Intel VNNI and bfloat16 instructions respectively) which have been shown to sufficiently cover a range of DL training and inference workloads [30], [31], [32], [33]…”
MethodSelf-citationSENeCA: Scalable Energy-efficient Neuromorphic Computer Architecture
Yousefzadeh, A., van Schaik, G., Tahghighi, M., Detterer, P., Traferro, S., Hijdra, M., Stuijt, J., Corradi, F., Sifalakis, M., & Konijnenburg, M. (2022) · International Conference on Artificial Intelligence Circuits and Systems · doi:10.1109/aicas54282.2022.9870025 · Cited by 29
“BF16 is a format which is initially suggested by Google to address the narrow dynamic range of IEEE FP16 [9].”
BackgroundETA: An Efficient Training Accelerator for DNNs Based on Hardware-Algorithm Co-Optimization
Lu, J., Ni, C., & Wang, Z. (2022) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2022.3145850 · Cited by 32 · see citing works
“1) Low-bit precision numbers, such as 16-bit floating-point (FP16) [24], [25], 8-bit integer (INT8) [26]–[28], and”
MethodHardware Trends Impacting Floating-Point Computations In Scientific Applications
Dongarra, J. J., Gunnels, J., Bayraktar, H., Haidar, A., & Ernst, D. (2024) · arXiv.org · doi:10.48550/arxiv.2411.12090 · Cited by 28
“Bfloat16 (BF16), which was introduced by Google [2], has the same range as FP32 at the expense of mantissa bits.”
A Unified Parallel CORDIC-Based Hardware Architecture for LSTM Network Acceleration
Mohamed, N. A., & Cavallaro, J. R. (2023) · IEEE transactions on computers · doi:10.1109/tc.2023.3268400 · Cited by 28 · see citing works
“Reduced bit-width floating-point formats such as IEEE’s halfprecision (float16), Google’s brain floating-point (bfloat16), and Nvidia’s 18-bit TensorFloat format have been shown to run effectively without accuracy loss on a variety of platforms [15], [16].”
BackgroundContrastive Tuning: A Little Help to Make Masked Autoencoders Forget
Lehner, J., Alkin, B., Furst, A., Rumetshofer, E., Miklautz, L., & Hochreiter, S. (2023) · AAAI Conference on Artificial Intelligence · doi:10.48550/arxiv.2304.10520 · Cited by 28
“8 using bfloat16 [51] precision and FlashAttention [22].”
MethodMultiple-Mode-Supporting Floating-Point FMA Unit for Deep Learning Processors
Tan, H., Tong, G., Huang, L.-B., Xiao, L.-Q., & Xiao, N. (2023) · IEEE Transactions on Very Large Scale Integration (vlsi) Systems · doi:10.1109/tvlsi.2022.3226185 · Cited by 27 · see citing works
“In light of increasing attempts to reduce the memory footprint and maintain enough precision for DNN training and inference, mixed-precision operations are becoming more popular in the literature [19], [23], [24], [25], [26].”
BackgroundMethodInfluentialDarKnight: A Data Privacy Scheme for Training and Inference of Deep Neural NetworksPreprint
Hashemi, H., Wang, Y., & Annavaram, M. (2020) · arXiv.org · doi:10.48550/arxiv.2006.01300 · Cited by 27 · see citing works
An Efficient Training Accelerator for Transformers With Hardware-Algorithm Co-Optimization
Shao, H., Lu, J., Wang, M., & Wang, Z. (2023) · IEEE Transactions on Very Large Scale Integration (VLSI) Systems · doi:10.1109/tvlsi.2023.3305569 · Cited by 21 · see citing works
“Other specific data formats, such as block floating point (BFP) [31] and brain floating point (BFLOAT) [32], have also been applied for efficient training.”
BackgroundOscillation-Reduced MXFP4 Training for Vision Transformers
Chen, Y.-X., Xi, H.-C., Zhu, J., & Chen, J.-F. (2025) · International Conference on Machine Learning · doi:10.48550/arxiv.2502.20853 · Cited by 25
“While BF16 and FP16 precision remain the most widely used formats for deep learning training (Narang et al., 2017; Kalamkar et al., 2019), FP8 training (Sun et al., 2019; Micikevicius et al., 2022; NVIDIA, 2024c; Xi et al., 2024a) is becoming increasingly mature in these years, with successful…”
Spectra: Surprising Effectiveness of Pretraining Ternary Language Models at ScalePreprint
Kaushal, A., Pandey, T., Vaidhya, T., Mondal, A. K., Bhagat, A., & Rish, I. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2407.12327 · Cited by 25 · see citing works
“Several prominent language models such as GPT [Brown et al., 2020b], NeoX [Black et al., 2022], Llama and Pythia families have been traditionally trained using mixed precision (FP32/FP16 or FP32/BF16) [Micikevicius et al., 2018] or half-precision (FP16/BF16) [Kalamkar et al., 2019].”
MethodTOPLOC: A Locality Sensitive Hashing Scheme for Trustless Verifiable Inference
Ong, J. M., Ferrante, M. D., Pazdera, A., Garner, R., Jaghouar, S., Basra, M., Ryabinin, M., & Hagemann, J. (2025) · International Conference on Machine Learning · doi:10.48550/arxiv.2501.16007 · Cited by 24
“Inference providers might use lower precision formats, such as fp8 (Micikevicius et al., 2022) or bf16 (Kalamkar et al., 2019), which significantly reduces the inference compute and memory requirements.”
NeuZip: Memory-Efficient Training and Inference with Dynamic Compression of Neural Networks
Hao, Y.-C., Cao, Y.-S., & Mou, L.-L. (2024) · arXiv.org · doi:10.48550/arxiv.2410.20650 · Cited by 24
Designing Efficient and High-Performance AI Accelerators With Customized STT-MRAM
Mishty, K., & Sadi, M. (2021) · IEEE Transactions on Very Large Scale Integration (vlsi) Systems · doi:10.48550/arxiv.2104.02199 · Cited by 24
“For inference-only accelerator int8 datatype and hardware suffice, however, if full-scale training or transfer-learning is desired then BF16 hardware and data-type are necessary [19], [20].”
Cheetah: Mixed Low-Precision Hardware & Software Co-Design Framework for DNNs on the EdgePreprint
Langroudi, H. F., Carmichael, Z., Pastuch, D., & Kudithipudi, D. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1908.02386 · Cited by 21 · see citing works
“To prevent the requirement of the loss scaling in mixed-precision floating point, Kalamkar et al. [37] proposed the brain floating point (BFLOAT-16) half-precision format with similar dynamic range (7-bit exponent) and less precision (8-bit fraction) compared to 32-bit floating point.”
Shifted and Squeezed 8-bit Floating Point format for Low-Precision Training of Deep Neural Networks
Cambier, L., Bhiwandiwalla, A., Gong, T., Nekuii, M., Elibol, O. H., & Tang, H. (2020) · arXiv · doi:10.48550/arxiv.2001.05674 · Cited by 23 · see citing works
“However, 8-bit precision training remains an open challenge (Johnson, 2018; Kalamkar et al., 2019).”
BackgroundWhich Prompts Make The Difference? Data Prioritization For Efficient Human LLM Evaluation
Boubdir, M., Kim, E., Ermi̇ş, B., Fadaee, M., & Hooker, S. (2023) · arXiv.org · doi:10.48550/arxiv.2310.14424 · Cited by 23
“To reduce memory usage, we run inference in a Bfloat16 setting [37].”
MethodBiLD: Bi-directional Logits Difference Loss for Large Language Model Distillation
Li, M.-C., Zhou, F., & Song, X.-H. (2024) · International Conference on Computational Linguistics · doi:10.48550/arxiv.2406.13555 · Cited by 22
“To reduce memory usage, we employ DeepSpeed during both SFT and distillation processes, along with gradient checkpointing and BFLOAT16 mode (Kalamkar et al., 2019).”
MethodNadalini, D., Rusci, M., Benini, L., & Conti, F. (2023) · Future generations computer systems · doi:10.48550/arxiv.2305.19167 · Cited by 22
Unit Scaling: Out-of-the-Box Low-Precision Training
Blake, C., Orr, D., & Luschi, C. (2023) · International Conference on Machine Learning · doi:10.48550/arxiv.2303.11257 · Cited by 22
Lim, J. P., & Nagarakatte, S. (2022) · Proc. ACM Program. Lang. · doi:10.1145/3498664 · Cited by 22
“Intel’s Bfloat16 [Kalamkar et al. 2019] and FlexPoint [Köster et al.”
BackgroundZFP: A compressed array representation for numerical computations
Lindström, P., Hittinger, J., Diffenderfer, J., Fox, A., Osei-Kuffuor, D., & Banks, J. W. (2024) · The international journal of high performance computing applications · doi:10.1177/10943420241284023 · Cited by 21 · see citing works
“Interest in half precision Markidis et al. (2018) and BFLOAT Burgess et al. (2019); Kalamkar et al. (2019) has been driven by data science applications where there may be different trade-offs with respect to the number of bits used to represent the mantissa or exponent.”
- Authors
- Kalamkar, D., Mudigere, D., Mellempudi, N., Das, D., Banerjee, K., Avancha, S., Vooturi, D. T., Jammalamadaka, N., Huang, J., Yuen, H., Yang, J., Park, J., Heinecke, A., Georganas, E., Srinivasan, S., Kundu, A., Smelyanskiy, M., Kaul, B., & Dubey, P.
- Affiliations
- Meta (United States) (US)
- Intel (United States) (US)
- ORCID
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2019
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 1905.12322
- OpenAlex
- W2947629474
- Cited by
- 169
- References
- 43
- Citations by year
- 2026: 4 · 2025: 14 · 2024: 15 · 2023: 33 · 2022: 32 · 2021: 37 · 2020: 32 · 2019: 2
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Computational Theory and Mathematics › Numerical Methods and Algorithms
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Hardware and Architecture › Parallel Computing and Optimization Techniques
- Keywords
- bfloat16, half-precision floating-point, mixed-precision training, deep learning training, rounding modes, image classification, speech recognition, language modeling, generative neural networks, recommender systems
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
