The DOI matches its DataCite record; 34 of the 43 references it cites were found in a registry or an index.
A Study of BFLOAT16 for Deep Learning Training
Open accessAbstract
This paper presents the first comprehensive empirical study demonstrating the efficacy of the Brain Floating Point (BFLOAT16) half-precision format for Deep Learning training across image classification, speech recognition, language modeling, generative networks and industrial recommendation systems. BFLOAT16 is attractive for Deep Learning training for two reasons: the range of values it can represent is the same as that of IEEE 754 floating-point format (FP32) and conversion to/from FP32 is simple. Maintaining the same range as FP32 is important to ensure that no hyper-parameter tuning is required for convergence; e.g., IEEE 754 compliant half-precision floating point (FP16) requires hyper-parameter tuning. In this paper, we discuss the flow of tensors and various key operations in mixed precision training, and delve into details of operations, such as the rounding modes for converting FP32 tensors to BFLOAT16. We have implemented a method to emulate BFLOAT16 operations in Tensorflow, Caffe2, IntelCaffe, and Neon for our experiments. Our results show that deep learning training using BFLOAT16 tensors achieves the same state-of-the-art (SOTA) results across domains as FP32 tensors in the same number of iterations and with no changes to hyper-parameters.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“This precision offers significant speedups for BERT models while maintaining comparable accuracy to FP32.”
From A Survey of Techniques for Optimizing Transformer Inference (2023) · cites this paper as [164]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“All these formats discussed here offer several benefits over the traditional floating-point precision, enabling faster and more efficient inference with reduced memory bandwidth.”
From A Survey of Techniques for Optimizing Transformer Inference (2023) · cites this paper as [164]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The emerging numerical formats such as Bfloat16 [164], Microsoft Floating Point (MSFP) [165], Tensorfloat32 [166], and FP8 [167, 168] provide better computational efficiency and accuracy for several transformer workloads.”
From A Survey of Techniques for Optimizing Transformer Inference (2023) · cites this paper as [164]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · cited 5×
“When training ResNet-50 on Imagenet using the AVX512BF16 instructions, we achieved a Top-1 accuracy of 75.62%, which matches the current state-of-the-art performance.”
MethodInfluentialVery Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 114,148 · cited 1×
“The network uses a “VGG loss” function using a pre-trained VGG19 network [33].”
MethodImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155 · cited 5×
“Here we choose AlexNet (ILSVRC 2012) [22] and ResNet-50 (ILSVRC 2015) [18] as representative models for the BFLOAT16 evaluation.”
MethodInfluentialImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333 · cited 1×
“However, all the aforementioned methods are shown to work for smaller benchmark model/data-sets only and invariably result in a non-trivial drop in accuracy with larger ImageNet data-set [9] and classification task [32].”
MethodImageNet Large Scale Visual Recognition Challenge
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. (2015) · International Journal of Computer Vision · doi:10.1007/s11263-015-0816-y · Cited by 39,380 · cited 1×
“However, all the aforementioned methods are shown to work for smaller benchmark model/data-sets only and invariably result in a non-trivial drop in accuracy with larger ImageNet data-set [9] and classification task [32].”
MethodPhoto-Realistic Single Image Super-Resolution Using a Generative Adversarial Network
Ledig, C., Theis, L., Huszar, F., Caballero, J., Cunningham, A., Acosta, A., Aitken, A., Tejani, A., Totz, J., Wang, Z., et al. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.19 · Cited by 12,696
Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network
Ledig, C., Theis, L., Huszár, F., Caballero, J., Cunningham, A., Acosta, A., Aitken, A., Tejani, A., Totz, J., Wang, Z.-H., et al. (2016) · Computer Vision and Pattern Recognition · doi:10.48550/arxiv.1609.04802 · Cited by 12,624 · cited 5×
“Using multiple deep learning frameworks modified to insert appropriate Quantlib calls, we provide SOTA results for: AlexNet [22], ResNet-50 [18], DC-GAN [29], SR-GAN [24], DeepSpeech2 [4], GNMT [37], and two industrial workloads namely: a Deep and Cross Network, and a DNN Recommendation System.”
BackgroundMethodInfluentialUnsupervised representation learning with deep convolutional generative adversarial networks
Radford, A., Metz, L., & Chintala, S. (2015) · arXiv · doi:10.48550/arxiv.1511.06434 · Cited by 12,235 · cited 7×
“We have implemented DC-GAN in Caffe and for our experiments all the input tensors (activations, weights) are converted to BFLOAT16 for convolution layers (in both the generator and the discriminator), while only the input activations are converted to BFLOAT16 for batch normalization layers; all other tensors are maint…”
BackgroundMethodInfluentialTensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems
Abadi, M., Agarwal, A., Barham, P., Brevdo, E., Chen, Z., Citro, C., Corrado, G. S., Davis, A., Dean, J., Devin, M., et al. (2016) · arXiv · doi:10.48550/arxiv.1603.04467 · Cited by 9,408 · cited 2×
“There are at least three half-precision formats in the domain of mixed precision training of large neural networks: FP16 [28], 16-bit Integer based [9], and BFLOAT16 [10, 4, 3].”
MethodJulia: A Fresh Approach to Numerical ComputingPreprint
Bezanson, J., Edelman, A., Karpinski, S., & Shah, V. B. (2014) · SIAM Review · doi:10.48550/arxiv.1411.1607 · Cited by 7,077 · cited 1×
“The Julia language [5], which is designed to provide high-performance without sacrificing ease of programming, has also been shown to be benefited from BFLOAT16 [11].”
Background
Research works printed but not linked: 5
- [10][10] Jeffrey Dean, Greg Corrado, Rajat Monga, Kai Chen, Matthieu Devin, Mark Mao, Andrew Senior, Paul Tucker, Ke Yang, Quoc V Le, et al. Large scale distributed deep networks. In Advances in neural information processing systems , pages 1223-1231, 2012. -
- [15][15] Boris Ginsburg, Sergei Nikolaev, and Paulius Micikevicius. Training of deep networks with half-precision float. NVidia GPU Technology Conference , 2017. -
- [34][34] Matthew Richardson, Ewa Dominowska, and Robert Ragno. Predicting clicks: Estimating the click-through rate for new ads. In Proceedings of the 16th International Conference on World Wide Web , WWW'07, pages 521-530, New York, NY, USA, 2007. ACM. -
- [37][37] Vincent Vanhoucke, Andrew Senior, and Mark Z Mao. Improving the speed of neural networks on CPUs. In Proc. Deep Learning and Unsupervised Feature Learning NIPS Workshop , volume 1, 2011. -
- [43][43] Jian Zhang, Jiyan Yang, and Hector Yuen. Training with low-precision embedding tables. In Systems for Machine Learning Workshop at NeurIPS 2018 . 2018.
Cited by
Sorted by citations
Large Language Models in Finance: A Survey
Li, Y., Wang, S., Ding, H., & Chen, H. (2023) · ACM International Conference on AI in Finance (ICAIF) · doi:10.1145/3604237.3626869 · Cited by 412
“In Section 5, we provide a more detailed discussion of which option is more favorable under different circumstances.”
BackgroundTen Lessons From Three Generations Shaped Google’s TPUv4i : Industrial Product
Jouppi, N. P., Yoon, D. H., Ashcraft, M. B., Gottscho, M., Jablin, T. B., Kurian, G. T., Laudon, J. P., Li, S., Ma, P., Ma, X., et al. (2021) · Proceedings - International Symposium on Computer Architecture/Proceedings · doi:10.1109/isca52012.2021.00010 · Cited by 403
“Bfloat16 is a better match to DNNs than IEEE 754 fp16 [24], so the MXU was changed to become the first hardware to support it, with 16K MAC units (½ the side of the systolic array, so ¼ of the size).”
FP8 Formats for Deep LearningPreprint
Micikevicius, P., Stosic, D., Burgess, N., Cornea, M., Dubey, P., Grisenthwaite, R., Ha, S., Heinecke, A., Judd, P., Kamalu, J., et al. (2022) · arXiv.org · doi:10.48550/arxiv.2209.05433 · Cited by 352
“Common floating point types for training include IEEE single precision, TF32 mode for single precision [19], IEEE half precision [14], and bfloat16 [9].”
Self-citationA domain-specific supercomputer for training deep neural networks
Jouppi, N., Yoon, D., Kurian, G., Li, S., Patil, N., Laudon, J., Young, C., & Patterson, D. A. (2020) · Communications of the ACM · doi:10.1145/3360307 · Cited by 312
“Beyond our experience that it works for training production applications, a recent Intel study corroborated its benefits.(21) However, fp16 requires adjustments to training software (loss scaling) to deliver convergence and efficiency.”
MethodResultEfficient Processing of Deep Neural Networks
Sze, V., Chen, Y., Yang, T.-J., & Emer, J. (2020) · Synthesis lectures on computer architecture · doi:10.2200/s01004ed1v01y202004cac050 · Cited by 125
NVILA: Efficient Frontier Visual Language Models
Liu, Z., Zhu, L., Shi, B., Zhang, Z., Lou, Y., Yang, S. W., Xi, H., Cao, S., Gu, Y., Li, D., et al. (2025) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52734.2025.00390 · Cited by 190
“FP16 [31] and BF16 [32] are standard precisions for model training, since they offer acceleration without accuracy loss, supported natively by NVIDIA GPUs.”
The Deep Learning Compiler: A Comprehensive Survey
Li, M., Liu, Y., Liu, X., Sun, Q., You, X., Yang, H., Luan, Z., Gan, L., Yang, G., & Qian, D. (2020) · IEEE Transactions on Parallel and Distributed Systems · doi:10.1109/tpds.2020.3030548 · Cited by 293
“Besides, low-precision training and inference, including FP32, FP16, INT8, and non-IEEE floating-point format bfloat16 [45] are also supported.”
Degree-Quant: Quantization-Aware Training for Graph Neural NetworksPreprint
Tailor, S. A., Fernández-Marqués, J., & Lane, N. D. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2008.05000 · Cited by 136
“, 2017) and data formats (Carmichael et al., 2018; Kalamkar et al., 2019), it is 8-bit integer (INT8) quantization that has attracted the most attention.”
BackgroundEfficient Methods for Natural Language Processing: A Survey
Treviso, M., Lee, J.-U., Ji, T., van Aken, B., Cao, Q., Ciosici, M. R., Hassid, M., Heafield, K., Hooker, S., Raffel, C., et al. (2023) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00577 · Cited by 156
Reducing Transformer Key-Value Cache Size with Cross-Layer Attention
Brandon, W., Mishra, M., Nrusimha, A., Panda, R., & Ragan-Kelley, J. (2024) · Neural Information Processing Systems · doi:10.48550/arxiv.2405.12981 · Cited by 172
“We use mixed precision training [Micikevicius et al., 2018] in BF16 [Kalamkar et al., 2019] with gradient all-reduce and gradient accumulation in FP32 for training stability.”
MethodGranite Code Models: A Family of Open Foundation Models for Code IntelligencePreprint
Mishra, M., Stallone, M., Zhang, G., Shen, Y., Prasad, A. S., Soria, A. M., Merler, M., Selvam, P., Surendran, S., Singh, S., et al. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2405.04324 · Cited by 149
“We use Megatron’s distributed optimizer with mixed precision training (Micikevicius et al., 2018) in BF16 (Kalamkar et al., 2019) with gradient all-reduce and gradient accumulation in FP32 for training stability.”
MethodA Survey of Techniques for Optimizing Transformer Inference
Chitty-Venkata, K. T., Mittal, S., Emani, M., Vishwanath, V., & Somani, A. (2023) · Journal of systems architecture · doi:10.48550/arxiv.2307.07982 · Cited by 169
“formats such as Bfloat16 [158], Microsoft Floating Point (MSFP) [159], Tensorfloat32 [160], and FP8 [161, 162] provide better computational efficiency and accuracy for several transformer workloads.”
BackgroundInfluentialA Survey of Techniques for Optimizing Transformer Inference
Chitty-Venkata, K. T., Mittal, S., Emani, M., Vishwanath, V., & Somani, A. K. (2023) · Journal of Systems Architecture · doi:10.1016/j.sysarc.2023.102990 · Cited by 154
An Overview of Neural Network CompressionPreprint
O’Neill, J. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.03669 · Cited by 149
KERPLE: Kernelized Relative Positional Embedding for Length ExtrapolationPreprint
Chi, T.-C., Fan, T.-H., Ramadge, P. J., & Rudnicky, A. I. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2205.09921 · Cited by 14
DART-Math: Difficulty-Aware Rejection Tuning for Mathematical Problem-Solving
Tong, Y., Zhang, X., Wang, R., Wu, R., & He, J. (2024) · Neural Information Processing Systems · doi:10.48550/arxiv.2407.13690 · Cited by 118
“\n\n###Instruction:\n{query}\n\n### Response:\n Other Details: For efficiency, We utilize various tools / libraries / techniques including: • the DeepSpeed distributed framework (Rasley et al., 2020) with ZeRO (Rajbhandari et al., 2020) stage 3 • gradient checkpointing (Chen et al., 2016) • torch.compile (Ansel et al.…”
LN3Diff++: Scalable L atent N eural Fields D iffusion for Speedy 3D Generation
Lan, Y., Hong, F., Zhou, S., Yang, S., Meng, X., Chen, Y., Lyu, Z., Dai, B., Pan, X., & Loy, C. C. (2025) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2025.3633073 · Cited by 10
“For efficient and scalable training, we leverage BFloat16 [52] with FlashAttention [18], [19] enabled.”
Filtering, Distillation, and Hard Negatives for Vision-Language Pre-Training
Radenović, F., Dubey, A., Kadian, A., Mihaylov, T., Vandenhende, S., Patel, Y., Wen, Y., Ramanathan, V., & Mahajan, D. K. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00673 · Cited by 56
Dave, S., Baghdadi, R., Nowatzki, T., Avancha, S., Shrivastava, A., & Li, B. (2021) · Proceedings of the IEEE · doi:10.1109/jproc.2021.3098483 · Cited by 120
Self-citationWhy Do We Need Weight Decay in Modern Deep Learning?
D'Angelo, F., Andriushchenko, M., Varre, A., & Flammarion, N. (2023) · Neural Information Processing Systems · doi:10.48550/arxiv.2310.04415 · Cited by 107
“Using bfloat16 training helps to significantly speed up training and reduce the GPU memory requirements allowing to train larger models and use larger batches (Kalamkar et al., 2019).”
BackgroundA TinyML Platform for On-Device Continual Learning With Quantized Latent Replays
Ravaglia, L., Rusci, M., Nadalini, D., Capotondi, A., Conti, F., & Benini, L. (2021) · IEEE Journal on Emerging and Selected Topics in Circuits and Systems · doi:10.1109/jetcas.2021.3121554 · Cited by 105
“The cluster includes a set of four highly flexible FPUs shared between all nine cores, capable of FP32 and FP16 computation [57].”
MethodPredicting cellular responses to perturbation across diverse contexts with State
Adduri, A. K., Gautam, D., Bevilacqua, B., Naghipourfar, M., Imran, A., Shah, R., Teyssier, N., Verma, R., Carpenter, C., Eraslan, B., et al. (2026) · Cell · doi:10.1016/j.cell.2026.07.052 · Cited by 100
With Shared Microexponents, A Little Shifting Goes a Long Way
Rouhani, B. D., Zhao, R., Elango, V., Shafipour, R., Hall, M., Mesmakhosroshahi, M., More, A., Melnick, L., Golub, M., Varatkar, G. V., et al. (2023) · Proceedings - International Symposium on Computer Architecture/Proceedings · doi:10.1145/3579371.3589351 · Cited by 81
Self-citationSecuring Approximate Homomorphic Encryption Using Differential Privacy
Li, B.-Y., Micciancio, D., Schultz, M., & Sorrell, J. (2022) · IACR Cryptology ePrint Archive · doi:10.1007/978-3-031-15802-5_20 · Cited by 96
“Some applications may solely require 8 or 16 bits of precision (see for example [16] or [33] for work on training ML models with low-precision computations).”
PANTHER: A Programmable Architecture for Neural Network Training Harnessing Energy-Efficient ReRAM
Ankit, A., Hajj, I. E., Chalamalasetti, S. R., Agarwal, S., Marinella, M., Foltin, M., Strachan, J. P., Milojicic, D., Hwu, W., & Roy, K. (2019) · IEEE transactions on computers · doi:10.1109/tc.2020.2998456 · Cited by 87
Lu, D., Wu, H., Liang, J., Xu, Y., He, Q., Geng, Y., Han, M., Xin, Y., & Xiao, Y. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2302.09432 · Cited by 95
“In particular, we found that using the BFLOAT16 (Kalamkar et al., 2019) half-precision floating-point format for optimization can effectively solve the problem of gradient overflow that occurs in the training process with FP16 halfprecision floating-point format, without the need to repeatedly…”
BackgroundMethodInfluentialTowards Cross-Tokenizer Distillation: the Universal Logit Distillation Loss for LLMsPreprint
Boizard, N., Haddad, K. E., Hudelot, C., & Colombo, P. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2402.12030 · Cited by 48
Defeating the Training-Inference Mismatch via FP16
Qi, P., Liu, Z.-C., Zhou, X.-X., Pang, T.-Y., Du, C., Lee, W.-S., & Lin, M. (2025) · arXiv.org · doi:10.48550/arxiv.2510.26788 · Cited by 78
“We find that merely switching the training precision from the now-dominant BF16 format [Dean et al., 2012, Kalamkar et al., 2019] to the earlier Float16 (FP16) format [Micikevicius et al., 2017] substantially mitigates the policy mismatch and yields significant performance improvements across RL…”
Park, G., Park, B., Kim, M., Lee, S., Kim, J.-H., Kwon, B., Kwon, S. J., Kim, B., Lee, Y., & Lee, D. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2206.09557 · Cited by 158
“As an example, BFLOAT16 [47] has 16 bits to represent a parameter, and thus, saves memory footprint and area overhead of arithmetic units compared to FP32.”
ResultKwon, D.-H., Lee, S., Kim, K., Oh, S., Park, J., Hong, G., Ka, D., Hwang, K.-D., Park, J.-J., Kang, K., et al. (2023) · IEEE Journal of Solid-State Circuits · doi:10.1109/jssc.2022.3200718 · Cited by 75
“The weight and the vector are bfloat16 (Brain Floating Point, BF16) format; it consists of 1-b sign (S), 8-b exponent (EX), and 7-b mantissa (MA) [6].”
BackgroundSelf-citationImproving Neural Network Efficiency via Post-training Quantization with Adaptive Floating-Point
Liu, F., Zhao, W., He, Z., Wang, Y., Wang, Z., Dai, C., Liang, X., & Jiang, L. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00523 · Cited by 68
“Specifically, FP32 format has an 8-bit exponent and 23-bit mantissa, FP16 has a 5-bit exponent and 10-bit mantissa, BF16 [16] has an 8-bit exponent and 7-bit mantissa, and TF32 has an 8-bit exponent and 10-bit mantissa.”
BackgroundMethodEvaluations on Deep Neural Networks Training Using Posit Number System
Lu, J., Fang, C., Xu, M., Lin, J. T., & Wang, Z. (2020) · IEEE Transactions on Computers · doi:10.1109/tc.2020.2985971 · Cited by 79
“Besides, the modified FP16 (BFLOAT16) format [16] was studied for DNN training.”
MethodEfficient training of large language models on distributed infrastructures: a survey
Duan, J., Zhang, S., Wang, Z., Jiang, L., Qu, W., Hu, Q., Wang, G., Weng, Q., Yan, H., Zhang, X., et al. (2026) · Vicinagearth. · doi:10.1007/s44336-026-00038-z · Cited by 84
“By employing lazy softmax, the division by j e s j in softmax can be delayed to the very end of the attention operation.”
Jaques, S., & Rattew, A. G. (2025) · Quantum · doi:10.22331/q-2025-12-02-1922 · Cited by 103
TensorDash: Exploiting Sparsity to Accelerate Deep Neural Network Training
Mahmoud, M., Edo, I., Zadeh, A. H., Awad, O. M., Pekhimenko, G. G., Albericio, J., & Moshovos, A. (2020) · IEEE/ACM International Symposium on Microarchitecture (MICRO) · doi:10.1109/micro50266.2020.00069 · Cited by 78
“bfloat16 [26], [41], [73], dynamic floating-point [14], and flexpoint [42]).”
MethodFitzGerald, J., Ananthakrishnan, S., Arkoudas, K., Bernardi, D., Bhagia, A., Delli Bovi, C., Cao, J., Chada, R., Chauhan, A., Chen, L., et al. (2022) · Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining · doi:10.1145/3534678.3539173 · Cited by 68
“Another way to mitigate the stability issues is to use BFLOAT16 [18], which was not available in Deepspeed at the time of our experiments.”
MethodFine Tuning LLM for Enterprise: Practical Guidelines and Recommendations
MathavRaj, J., Kushala, V., Warrier, H., & Gupta, Y. (2024) · arXiv.org · doi:10.48550/arxiv.2404.10779 · Cited by 66
“Brain float 16[15] or BF16 maintains the same range as FP32 but reduction of precision to 7 bits.”
BackgroundExploring the Role of Large Language Models in Prompt Encoding for Diffusion Models
Ma, B., Zong, Z.-F., Song, G.-L., Li, H.-S., & Liu, Y. (2024) · Neural Information Processing Systems · doi:10.48550/arxiv.2406.11831 · Cited by 61
“Besides, considering the broader numerical representation range of bfloat16, we finally use the bfloat16 mixed precision training [24] strategy.”
Michaud, E. J., Liu, Z., & Tegmark, M. (2023) · Entropy · doi:10.3390/e25010175 · Cited by 58
“Accordingly, to save memory and speed up computation, much work has gone into reducing the numerical precision used in models without sacrificing model performance much [1, 2, 3].”
Intriguing Properties of Quantization at ScalePreprint
Ahmadian, A., Dash, S., Chen, H.-Y., Venkitesh, B., Gou, S., Blunsom, P., Ustun, A., & Hooker, S. (2023) · Neural Information Processing Systems · doi:10.48550/arxiv.2305.19268 · Cited by 56
“In this technique, a copy of weights is stored in full-precision ( fp32 ) whereas the forward and backward passes are done using half-precision in either float16 ( fp16 ) or bfloat16 ( bf16 ) (Kalamkar et al., 2019; Dean et al., 2012; Abadi et al., 2015).”
MethodDefensive approximation: securing CNNs using approximate computing
Guesmi, A., Alouani, I., Khasawneh, K. N., Baklouti, M., Frikha, T., Abid, M., & Abu-Ghazaleh, N. (2021) · International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) · doi:10.1145/3445814.3446747 · Cited by 55
“The Bfloat16 (Brain Floating Point) [28] is a truncated version of the 32-bit IEEE 754 single-precision floating-point format (float32).”
BackgroundMaji, S., Banerjee, U., & Chandrakasan, A. P. (2021) · IEEE Internet of Things Journal · doi:10.1109/jiot.2021.3061314 · Cited by 61
Optimizing Deep Learning Recommender Systems Training on CPU Cluster Architectures
Kalamkar, D. D., Georganas, E., Srinivasan, S., Chen, J., Shiryaev, M., & Heinecke, A. (2020) · International Conference for High Performance Computing, Networking, Storage and Analysis · doi:10.1109/sc41405.2020.00047 · Cited by 52
“Classic training approaches using BFLOAT16, FP16 or even int16 require so-called master-weights, a full precision FP32 copy of the entire model throughout the full training process [34]–[36].”
Self-citationIntel Nervana Neural Network Processor-T (NNP-T) Fused Floating Point Many-Term Dot Product
Hickmann, B. J., Chen, J., Rotzin, M., Yang, A., Urbanski, M., & Avancha, S. (2020) · IEEE Symposium on Computer Arithmetic · doi:10.1109/arith48897.2020.00029 · Cited by 51
“…properties, The first major design decision was to select BFloat16 (BF16) as the numerical format over several alternatives such as FP32, FP16, and various integer formats, as it offered the power and area efficiency of a 16-bit format with convergence characteristics similar to that of FP32 [6].”
MethodSelf-citationA TensorFlow Simulation Framework for Scientific Computing of Fluid Flows on Tensor Processing Units
Wang, Q., Ihme, M., Chen, Y.-F., & Anderson, J. R. (2021) · Computer Physics Communications · doi:10.1016/j.cpc.2022.108292 · Cited by 50
“Software emulation with multiple passes of the MXU operations makes higher-precision matrix multiplication (32-bit and 64-bit) possible, with a performance trade-off [15].”
Lee, J., Mukhanov, L., Molahosseini, A. S., Minhas, U. I., Yang, H., del Rincón, J. M., Dichev, K., Hong, C.-H., & Vandierendonck, H. (2023) · ACM Computing Surveys · doi:10.1145/3587095 · Cited by 63
“A conventional mixed-precision training applied lowerprecision arithmetic to the multiplications in MACs, including both forward and backward paths, and higher-precision arithmetic to the accumulations in the MACs using lower-precision quantized operands [95, 125].”
MethodFastFold: Reducing AlphaFold Training Time from 11 Days to 67 HoursPreprint
Cheng, S., Zhao, X., Lu, G., Fang, J., Yu, Z., Zheng, T., Wu, R., Zhang, X., Peng, J., & You, Y. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2203.00854 · Cited by 34
“To speed up training time and reduce memory consumption, AlphaFold training uses Bfloat16 precision [9].”
MethodReasoning-to-Defend: Safety-Aware Reasoning Can Defend Large Language Models from Jailbreaking
Zhu, J., Yan, L., Wang, S.-Q., Yin, D.-W., & Sha, L. (2025) · Conference on Empirical Methods in Natural Language Processing · doi:10.48550/arxiv.2502.12970 · Cited by 47
“We conduct Distributed Data Parallel (DDP, Li et al., 2020) with mixed-precision training with bfloat16 data type (Kalamkar et al., 2019) implemented in apex 4 .”
COAT: Compressing Optimizer states and Activation for Memory-Efficient FP8 TrainingPreprint
Xi, H., Cai, H., Zhu, L., Lu, Y., Keutzer, K., Chen, J., & Han, S. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2410.19313 · Cited by 29
“BF16 training (Kalamkar et al., 2019) refines this approach, as BF16 has a larger representation range and is more stable for large-scale training.”
BackgroundMethodRethinking Data Selection for Supervised Fine-TuningPreprint
Shen, M. (2024) · arXiv.org · doi:10.48550/arxiv.2402.06094 · Cited by 44
“All models are trained with BFloat16 (Kalamkar et al., 2019) mixed precision for stability.”
MethodE-prop on SpiNNaker 2: Exploring online learning in spiking RNNs on neuromorphic hardware
Rostami, A., Vogginger, B., Yan, Y., & Mayr, C. (2022) · Frontiers in Neuroscience · doi:10.3389/fnins.2022.1018006 · Cited by 44
“, 2022) or mixedprecision training (lower precision data format for some variables) (Micikevicius et al., 2018; Kalamkar et al., 2019), lowrank matrix representation (Kusupati et al.”
BackgroundMixed-Precision Embedding Using a CachePreprint
Yang, J., Huang, J., Park, J., Tang, P. T. P., & Tulloch, A. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.11305 · Cited by 35
“Notable works using this approach include (Gupta et al., 2015; Chen et al., 2017; De Sa et al., 2018; Wu et al., 2018b; Zhang et al., 2018; Kalamkar et al., 2019).”
Self-citationSENECA: building a fully digital neuromorphic processor, design trade-offs and challenges
Tang, G., Vadivel, K., Xu, Y., Bilgic, R., Shidqi, K., Detterer, P., Traferro, S., Konijnenburg, M., Sifalakis, M., van Schaik, G.-J., et al. (2023) · Frontiers in Neuroscience · doi:10.3389/fnins.2023.1187252 · Cited by 40
“They also support shared scale factors (Köster et al., 2017; Moons et al., 2017; Jacob et al., 2018; Kalamkar et al., 2019; Coelho et al., 2021).”
BackgroundSAVE: Sparsity-Aware Vector Engine for Accelerating DNN Training and Inference on CPUs
Gong, Z., Ji, H., Fletcher, C. W., Hughes, C. J., Baghsorkhi, S. S., & Torrellas, J. (2020) · IEEE/ACM International Symposium on Microarchitecture (MICRO) · doi:10.1109/micro50266.2020.00070 · Cited by 40
“Training in BF16 yields an accuracy comparable to that of using FP32, without tuning hyperparameters [39], [65].”
BackgroundSearching for Winograd-aware Quantized NetworksPreprint
Fernández-Marqués, J., Whatmough, P. N., Mundy, A., & Mattina, M. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2002.10711 · Cited by 28
Four Over Six: More Accurate NVFP4 Quantization with Adaptive Block ScalingPreprint
Cook, J., Guo, J., Xiao, G., Lin, Y., Wyss, K., Nazemi, M., Mishra, A., del Mundo, C., Blankevoort, T., & Han, S. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2512.02010 · Cited by 41
“An example of this can be seen in numerical precision: it used to be standard to train models using FP32, then FP16 [1], and now it is standard to keep most or all parameters in BF16 [2].”
Zhou, Z., Li, C., Wei, X., Wang, X., & Sun, G. (2022) · International Conference on Parallel Architectures and Compilation Techniques (PACT) · doi:10.1145/3559009.3569670 · Cited by 46
“BF16 has the same accuracy as FP32 for NN training [47] but is more cost-efficient.”
MethodEvaluation Method of Deep Learning-Based Embedded Systems for Traffic Sign Detection
Lopez-Montiel, M. A., Orozco-Rosas, U., Sánchez-Adame, M., Picos, K., & Montiel, O. (2021) · IEEE Access · doi:10.1109/access.2021.3097969 · Cited by 55
“A regular TPU Cloud Service has two 128 × 128 systolic arrays, adding 32,768 ALUs for 16-bit floating-point values on a single processor [72].”
BackgroundZhang, T., Hariri, M., Sui, Y., Zhong, S.-C., Chaudhary, V., Hu, X. B., & Shrivastava, A. (2025) · Neural Information Processing Systems · doi:10.48550/arxiv.2504.11651 · Cited by 40
“Compared to FP16, BF16 offers a wider dynamic range at the cost of precision, which improves numerical stability and mitigates overflow issues during training [8, 19].”
Vision-LSTM: xLSTM as Generic Vision Backbone
Alkin, B., Beck, M., Pöppel, K., Hochreiter, S., & Brandstetter, J. (2024) · arXiv · doi:10.48550/arxiv.2406.04303 · Cited by 39
Multi-lingual Evaluation of Code Generation Models
Athiwaratkun, B., Gouda, S. K., Wang, Z., Li, X., Tian, Y., Tan, M., Ahmad, W. U., Wang, S., Sun, Q., Shang, M., et al. (2022) · arXiv · doi:10.48550/arxiv.2210.14868 · Cited by 38
“Our training pipeline is based on PyTorch Lightning § and we use bfloat16 (Kalamkar et al., 2019) and DeepSpeed (Rasley et al.”
MethodLearning compositional functions via multiplicative weight updatesPreprint
Bernstein, J., Zhao, J., Meister, M., Liu, M.-Y., Anandkumar, A., & Yue, Y. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.14560 · Cited by 14
Impact of Mixed Precision Techniques on Training and Inference Efficiency of Deep Neural Networks
Dörrich, M., Fan, M., & Kist, A. M. (2023) · IEEE Access · doi:10.1109/access.2023.3284388 · Cited by 43
“Moreover, it was shown that bfloat16 training of neural networks can achieve the same state-of-the-art results as the standard 32-bit format [10].”
BackgroundVarious Lengths, Constant Speed: Efficient Language Modeling with Lightning AttentionPreprint
Qin, Z., Sun, W., Li, D., Shen, X., Sun, W., & Zhong, Y. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2405.17381 · Cited by 22
Meteor: Mamba-based Traversal of Rationale for Large Language and Vision Models
Lee, B.-K., Kim, C. W., Park, B., & Yong, M.-R. (2024) · Neural Information Processing Systems · doi:10.48550/arxiv.2405.15574 · Cited by 37
Training and inference of large language models using 8-bit floating pointPreprint
Perez, S. P., Zhang, Y., Briggs, J., Blake, C., Levy-Kramer, J., Balanca, P., Luschi, C., Barlow, S. H., & Fitzgibbon, A. W. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2309.17224 · Cited by 23
“Models of more than 100 billion parameters like Bloom [Scao et al., 2022] or OPT [Zhang et al., 2022] struggled to find a stable loss scaling for FP16 and ended up employing BF16.”
MethodInfluentialDaghaghi, S., Meisburger, N., Zhao, M., Wu, Y., Gobriel, S., Tai, C., & Shrivastava, A. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2103.10891 · Cited by 21
“Brain floating-point format (BF16) uses 16 bits in computer memory to represent a floating-point number (Kalamkar et al., 2019).”
BackgroundUnlocking the Power of Inline Floating-Point Operations on Programmable Switches
Yuan, Y., Alama, O., Sapio, A., Fei, J., Nelson, J., Ports, D. R. K., Canini, M., & Kim, N. S. (2021) · King Abdullah University of Science and Technology Repository (King Abdullah University of Science and Technology) · doi:10.48550/arxiv.2112.06095 · Cited by 21
Revisiting BFloat16 TrainingPreprint
Zamirai, P., Zhang, J., Aberger, C. R., & De, C. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.06192 · Cited by 39
“State-of-the-art low-precision generic training algorithms use mixed precision, still requiring accelerators to contain 32-bit compute units [7, 8, 42].”
BackgroundMethodATM: Adversarial Tuning Multi-agent System Makes a Robust Retrieval-Augmented Generator
Zhu, J., Yan, L., Shi, H.-B., Yin, D.-W., & Sha, L. (2024) · Conference on Empirical Methods in Natural Language Processing · doi:10.48550/arxiv.2405.18111 · Cited by 35
“To this end, we use mixed-precision training with a model in bfloat16 data type (Kalamkar et al., 2019) implemented in apex and ZeRO (Rajbhandari et al., 2020) stage 1 implemented with DeepSpeed 8 which splits optimizer state across GPUs thus saving memory.”
MethodMixed-precision numerics in scientific applications: survey and perspectives
Kashi, A., Lu, H., Brewer, W., Rogers, D., Matheson, M. A., Shankar, M., & Wang, F.-Y. (2024) · Journal of Supercomputing · doi:10.48550/arxiv.2412.19322 · Cited by 34
“Brain Float 16, usually referred to as BFloat16 or BF16, was developed by Google specifically for deep learning training on their Tensor Processing Units (TPUs) [9], and has since become a widely adopted standard.”
First-Generation Inference Accelerator Deployment at FacebookPreprint
Anderson, M. J., Chen, B., Chen, S. Y., Deng, S., Fix, J., Gschwind, M. K., Kalaiah, A., Kim, C., Lee, J., Liang, J., et al. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2107.04140 · Cited by 29
“Since recent training hardware supports bfloat16 [29], [31], [43], bfloat16 support in inference accelerators can be useful to unify numerical format between”
BackgroundSelf-citationField-Programmable Gate Array Architecture for Deep Learning: Survey and Future Directions
Boutros, A., Arora, A., & Betz, V. (2024) · Proceedings of the IEEE · doi:10.1109/jproc.2025.3623023 · Cited by 32
“Each PE in the tensor block can implement 1 × int16 , fp16 , or 16-bit brain floating-point ( bfloat16 ) [129] MAC, and it can also be fractured to implement 4 × int8 MACs.”
An approach to generate correctly rounded math libraries for new floating point variants
Lim, J. P., Aanjaneya, M., Gustafson, J. L., & Nagarakatte, S. (2021) · Proc. ACM Program. Lang. · doi:10.1145/3434310 · Cited by 32
“Bfloat16 ’s dynamic range is similar to a 32-bit float but has lower memory traffic and footprint, which makes it appealing for neural networks [Kalamkar et al. 2019].”
BackgroundQA‐LIGN: Aligning LLMs through Constitutionally Decomposed QA
Dineen, J., Rrv, A., Liu, Q., Xu, Z., Ye, X., Shen, M., Li, Z., Lu, S., Baral, C. R., Chen, M., et al. (2025) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2025.findings-emnlp.1123 · Cited by 22
“All models were fine-tuned with a maximum sequence length of 1,536 tokens and run in BFloat16 mixed precision (Kalamkar et al., 2019)..”
Pre-training Distillation for Large Language Models: A Design Space Exploration
Peng, H., Lv, X., Bai, Y., Yao, Z.-J., Zhang, J.-J., Hou, L., & Li, J.-Z. (2024) · Annual Meeting of the Association for Computational Linguistics · doi:10.48550/arxiv.2410.16215 · Cited by 31
BioTrove: A Large Curated Image Dataset Enabling AI for Biodiversity
Yang, C.-H., Feuer, B., Jubery, T., Deng, Z. K., Nakkab, A., Hasan, M. Z. Z., Chiranjeevi, S., Marshall, K. O., Baishnab, N., Singh, A., et al. (2024) · Neural Information Processing Systems · doi:10.52202/079017-3241 · Cited by 31
“We train our models for a combined 10 days on 8xH100 nodes in bfloat16 precision [18] with gradient checkpointing, computing loss with local features, and utilizing static graph optimization for DDP.”
A Reconfigurable Processing Element for Multiple-Precision Floating/Fixed-Point HPC
Li, B.-Y., Li, K., Zhou, J.-J., Ren, Y., Mao, W., Yu, H., & Wong, N. (2024) · IEEE Transactions on Circuits and Systems - II - Express Briefs · doi:10.1109/tcsii.2023.3322259 · Cited by 31
“These parts also appear in BF16 [18] and TF32 [19], but different floating-point numbers have different exponent and mantissa bitwidths as well as exponent biases ( bias ) .”
BackgroundHarnessing Deep Learning via a Single Building Block
Georganas, E., Banerjee, K., Kalamkar, D. D., Avancha, S., Venkat, A., Anderson, M. R., Henry, G., Pabst, H., & Heinecke, A. (2020) · IEEE International Parallel and Distributed Processing Symposium (IPDPS) · doi:10.1109/ipdps47924.2020.00032 · Cited by 44
“The results in section IV are in single precision (FP32), however we already have implementations supporting the int8 and bfloat16 datatypes (via the new Intel VNNI and bfloat16 instructions respectively) which have been shown to sufficiently cover a range of DL training and inference workloads [30], [31], [32], [33]…”
MethodSelf-citationSENeCA: Scalable Energy-efficient Neuromorphic Computer Architecture
Yousefzadeh, A., van Schaik, G.-J., Tahghighi, M., Detterer, P., Traferro, S., Hijdra, M., Stuijt, J., Corradi, F., Sifalakis, M., & Konijnenburg, M. (2022) · IEEE International Conference on Artificial Intelligence Circuits and Systems (AICAS) · doi:10.1109/aicas54282.2022.9870025 · Cited by 24
“BF16 is a format which is initially suggested by Google to address the narrow dynamic range of IEEE FP16 [9].”
BackgroundETA: An Efficient Training Accelerator for DNNs Based on Hardware-Algorithm Co-Optimization
Lu, J., Ni, C., & Wang, Z. (2022) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2022.3145850 · Cited by 34
“1) Low-bit precision numbers, such as 16-bit floating-point (FP16) [24], [25], 8-bit integer (INT8) [26]–[28], and”
MethodHardware Trends Impacting Floating-Point Computations In Scientific Applications
Dongarra, J. J., Gunnels, J., Bayraktar, H., Haidar, A., & Ernst, D. (2024) · arXiv.org · doi:10.48550/arxiv.2411.12090 · Cited by 28
“Bfloat16 (BF16), which was introduced by Google [2], has the same range as FP32 at the expense of mantissa bits.”
A Unified Parallel CORDIC-Based Hardware Architecture for LSTM Network Acceleration
Mohamed, N. A., & Cavallaro, J. R. (2023) · IEEE Transactions on Computers · doi:10.1109/tc.2023.3268400 · Cited by 24
“Reduced bit-width floating-point formats such as IEEE’s halfprecision (float16), Google’s brain floating-point (bfloat16), and Nvidia’s 18-bit TensorFloat format have been shown to run effectively without accuracy loss on a variety of platforms [15], [16].”
BackgroundContrastive Tuning: A Little Help to Make Masked Autoencoders Forget
Lehner, J., Alkin, B., Furst, A., Rumetshofer, E., Miklautz, L., & Hochreiter, S. (2023) · AAAI Conference on Artificial Intelligence · doi:10.48550/arxiv.2304.10520 · Cited by 28
“8 using bfloat16 [51] precision and FlashAttention [22].”
MethodMultiple-Mode-Supporting Floating-Point FMA Unit for Deep Learning Processors
Tan, H., Tong, G., Huang, L.-B., Xiao, L.-Q., & Xiao, N. (2023) · IEEE Transactions on Very Large Scale Integration (vlsi) Systems · doi:10.1109/tvlsi.2022.3226185 · Cited by 27
“In light of increasing attempts to reduce the memory footprint and maintain enough precision for DNN training and inference, mixed-precision operations are becoming more popular in the literature [19], [23], [24], [25], [26].”
BackgroundMethodInfluentialDarKnight: A Data Privacy Scheme for Training and Inference of Deep Neural NetworksPreprint
Hashemi, H., Wang, Y., & Annavaram, M. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.01300 · Cited by 20
An Efficient Training Accelerator for Transformers With Hardware-Algorithm Co-Optimization
Shao, H., Lu, J., Wang, M., & Wang, Z. (2023) · IEEE Transactions on Very Large Scale Integration (VLSI) Systems · doi:10.1109/tvlsi.2023.3305569 · Cited by 24
“Other specific data formats, such as block floating point (BFP) [31] and brain floating point (BFLOAT) [32], have also been applied for efficient training.”
BackgroundOscillation-Reduced MXFP4 Training for Vision Transformers
Chen, Y.-X., Xi, H.-C., Zhu, J., & Chen, J.-F. (2025) · International Conference on Machine Learning · doi:10.48550/arxiv.2502.20853 · Cited by 25
“While BF16 and FP16 precision remain the most widely used formats for deep learning training (Narang et al., 2017; Kalamkar et al., 2019), FP8 training (Sun et al., 2019; Micikevicius et al., 2022; NVIDIA, 2024c; Xi et al., 2024a) is becoming increasingly mature in these years, with successful…”
Spectra: Surprising Effectiveness of Pretraining Ternary Language Models at ScalePreprint
Kaushal, A., Vaidhya, T., Mondal, A. K., Pandey, T., Bhagat, A., & Rish, I. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2407.12327 · Cited by 25
“Several prominent language models such as GPT [Brown et al., 2020b], NeoX [Black et al., 2022], Llama and Pythia families have been traditionally trained using mixed precision (FP32/FP16 or FP32/BF16) [Micikevicius et al., 2018] or half-precision (FP16/BF16) [Kalamkar et al., 2019].”
MethodTOPLOC: A Locality Sensitive Hashing Scheme for Trustless Verifiable Inference
Ong, J. M., Ferrante, M. D., Pazdera, A., Garner, R., Jaghouar, S., Basra, M., Ryabinin, M., & Hagemann, J. (2025) · International Conference on Machine Learning · doi:10.48550/arxiv.2501.16007 · Cited by 24
“Inference providers might use lower precision formats, such as fp8 (Micikevicius et al., 2022) or bf16 (Kalamkar et al., 2019), which significantly reduces the inference compute and memory requirements.”
NeuZip: Memory-Efficient Training and Inference with Dynamic Compression of Neural Networks
Hao, Y.-C., Cao, Y.-S., & Mou, L.-L. (2024) · arXiv.org · doi:10.48550/arxiv.2410.20650 · Cited by 24
Designing Efficient and High-Performance AI Accelerators With Customized STT-MRAM
Mishty, K., & Sadi, M. (2021) · IEEE Transactions on Very Large Scale Integration (vlsi) Systems · doi:10.48550/arxiv.2104.02199 · Cited by 24
“For inference-only accelerator int8 datatype and hardware suffice, however, if full-scale training or transfer-learning is desired then BF16 hardware and data-type are necessary [19], [20].”
Cheetah: Mixed Low-Precision Hardware & Software Co-Design Framework for DNNs on the EdgePreprint
Langroudi, H. F., Carmichael, Z., Pastuch, D., & Kudithipudi, D. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1908.02386 · Cited by 31
“To prevent the requirement of the loss scaling in mixed-precision floating point, Kalamkar et al. [37] proposed the brain floating point (BFLOAT-16) half-precision format with similar dynamic range (7-bit exponent) and less precision (8-bit fraction) compared to 32-bit floating point.”
Shifted and Squeezed 8-bit Floating Point format for Low-Precision Training of Deep Neural Networks
Cambier, L., Bhiwandiwalla, A., Gong, T., Nekuii, M., Elibol, O. H., & Tang, H. (2020) · arXiv · doi:10.48550/arxiv.2001.05674 · Cited by 23
“However, 8-bit precision training remains an open challenge (Johnson, 2018; Kalamkar et al., 2019).”
BackgroundWhich Prompts Make The Difference? Data Prioritization For Efficient Human LLM EvaluationPreprint
Boubdir, M., Kim, E., Ermiş, B., Fadaee, M., & Hooker, S. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2310.14424 · Cited by 17
“To reduce memory usage, we run inference in a Bfloat16 setting [37].”
MethodBiLD: Bi-directional Logits Difference Loss for Large Language Model Distillation
Li, M.-C., Zhou, F., & Song, X.-H. (2024) · International Conference on Computational Linguistics · doi:10.48550/arxiv.2406.13555 · Cited by 22
“To reduce memory usage, we employ DeepSpeed during both SFT and distillation processes, along with gradient checkpointing and BFLOAT16 mode (Kalamkar et al., 2019).”
MethodNadalini, D., Rusci, M., Benini, L., & Conti, F. (2023) · Future generations computer systems · doi:10.48550/arxiv.2305.19167 · Cited by 22
Unit Scaling: Out-of-the-Box Low-Precision TrainingPreprint
Blake, C., Orr, D., & Luschi, C. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2303.11257 · Cited by 6
Lim, J. P., & Nagarakatte, S. (2022) · Proc. ACM Program. Lang. · doi:10.1145/3498664 · Cited by 22
“Intel’s Bfloat16 [Kalamkar et al. 2019] and FlexPoint [Köster et al.”
BackgroundZFP: A compressed array representation for numerical computations
Lindström, P., Hittinger, J., Diffenderfer, J., Fox, A., Osei-Kuffuor, D., & Banks, J. W. (2024) · The international journal of high performance computing applications · doi:10.1177/10943420241284023 · Cited by 21
“Interest in half precision Markidis et al. (2018) and BFLOAT Burgess et al. (2019); Kalamkar et al. (2019) has been driven by data science applications where there may be different trade-offs with respect to the number of bits used to represent the mantissa or exponent.”
Metadata
- Authors
- Kalamkar, D., Mudigere, D., Mellempudi, N., Das, D., Banerjee, K., Avancha, S., Vooturi, D. T., Jammalamadaka, N., Huang, J., Yuen, H., Yang, J., Park, J., Heinecke, A., Georganas, E., Srinivasan, S., Kundu, A., Smelyanskiy, M., Kaul, B., & Dubey, P.
- Affiliations
- Meta (United States) (US)
- Intel (United States) (US)
- ORCID
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2019
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 1905.12322
- OpenAlex
- W2947629474
- Cited by
- 169
- References
- 43
- Citations by year
- 2026: 4 · 2025: 14 · 2024: 15 · 2023: 33 · 2022: 32 · 2021: 37 · 2020: 32 · 2019: 2
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Computational Theory and Mathematics › Numerical Methods and Algorithms
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Hardware and Architecture › Parallel Computing and Optimization Techniques
- Keywords
- bfloat16, half-precision floating-point, mixed-precision training, deep learning training, rounding modes, image classification, speech recognition, language modeling, generative neural networks, recommender systems
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
