A Study of BFLOAT16 for Deep Learning Training

“When training ResNet-50 on Imagenet using the AVX512BF16 instructions, we achieved a Top-1 accuracy of 75.62%, which matches the current state-of-the-art performance.”

“The network uses a “VGG loss” function using a pre-trained VGG19 network [33].”

“Here we choose AlexNet (ILSVRC 2012) [22] and ResNet-50 (ILSVRC 2015) [18] as representative models for the BFLOAT16 evaluation.”

“However, all the aforementioned methods are shown to work for smaller benchmark model/data-sets only and invariably result in a non-trivial drop in accuracy with larger ImageNet data-set [9] and classification task [32].”

“However, all the aforementioned methods are shown to work for smaller benchmark model/data-sets only and invariably result in a non-trivial drop in accuracy with larger ImageNet data-set [9] and classification task [32].”

“Using multiple deep learning frameworks modified to insert appropriate Quantlib calls, we provide SOTA results for: AlexNet [22], ResNet-50 [18], DC-GAN [29], SR-GAN [24], DeepSpeech2 [4], GNMT [37], and two industrial workloads namely: a Deep and Cross Network, and a DNN Recommendation System.”

“We have implemented DC-GAN in Caffe and for our experiments all the input tensors (activations, weights) are converted to BFLOAT16 for convolution layers (in both the generator and the discriminator), while only the input activations are converted to BFLOAT16 for batch normalization layers; all other tensors are maint…”

“There are at least three half-precision formats in the domain of mixed precision training of large neural networks: FP16 [28], 16-bit Integer based [9], and BFLOAT16 [10, 4, 3].”

“The Julia language [5], which is designed to provide high-performance without sacrificing ease of programming, has also been shown to be benefited from BFLOAT16 [11].”

“Alternatives such as BFloat16 (Burgess et al., 2019; Kalamkar et al., 2019), minifloats (Fox et al., 2020), logarithmic fixed-point numbers (Johnson, 2018, 2020; Sun et al., 2020), posits (Chaurasiya et al., 2018; Gustafson & Yonemoto, 2017; Klöwer et al., 2019, 2020; Langroudi et al., 2019; Zhang & Ko, 2020) and stoc…”

“Bfloat16 structure uses a similar approach to float and bfloat conversion, analyzing performance capabilities of low-precision bfloat in deep learning networks [14].”

“A higher value of EPM indicates better preservation of the original (BFloat16 [18]) model performance.”

“When the model with parameter size ϕ and uses FP16/BF16 mixed-precision training (Micikevicius et al., 2017; Kalamkar et al., 2019) with the Adam optimizer (Kingma, 2014), the parameters and gradients require 4 ϕ bytes of memory, while the optimizer states require 12 ϕ bytes.”

“Each instruction is executed in one cycle (pipelined, 2ns per cycle) and operates in BrainFloat 16 (BF16) format [15].”

“To increase the numerical range of the IEEE float-16 format, the DLFloat format with 6 exponent bits and 9 significand bits (Agrawal et al., 2019) and the BFloat-16 format with 8 exponent bits and 7 significand bits (Kalamkar et al., 2019) have been proposed for training deep neural networks.”

“The impact on deep learning application of using lower-precision datatypes, supported by Matrix Cores, is widely studied, such as IEEE half-precision [30], bfloat16 [31], along with integer datatypes, such as int8 [32].”

“In addition, some frequently used routines like Weight Update (WU) operations or Batch Normalization (BN) must entirely rely on the FP32 format to achieve state-of-the-art accuracy [6], [7].”

“[92] to avoid the entailment of loss scaling in [91].”

“Furthermore, dynamic mixed precision training saves checkpoints in one data format (e.g., float16) and switches to another one (e.g., bfloat16 or float32) at different training stages, which have different bit allocation policies between the mantissa and exponent [25].”

“Training using 16-bit floating-point formats such as FP16 (Narang et al., 2017) or BF16 (Kalamkar et al., 2019), as well as 8-bit formats such as FP8 (Liu et al., 2024; Peng et al., 2023) is increasingly mature and has reached commercial deployment (Liu et al., 2024), and these efforts are enabled in part by modern ha…”

“Its use reduces memory bandwidth requirements without significantly impacting model accuracy(Kalamkar et al., 2019), making it ideal for managing the computational demands of long-context models.”

“Other, complimentary work focuses uniquely on the DSA [11] or data center regulation [16].”

“The resulting fused GEMM blocks yield floating-point precision outputs, namely FP16, BF16 or FP32, depending on the selected format.”

“Measurements are extracted on an isolated compute node with four Tesla A100 80GB and bf16 mixed precision [28].”

“Similar to GS-LRM, we apply the common practice to save GPU VRAMs using xFormers [41], deferred backpropagation [101], gradient checkpointing [12], and BF16 mixed-precision training [36].”

“Mixed-precision computation is extensively employed in DNN’s training [9], [12] and iteratively refining matrix equation solvers [8], [24] for enhanced performance and overhead reduction.”

“One such method is mixed precision training [23], where the multiplications are performed in a lower precision format such as float16 or bfloat16 [15] while the accumulation is still performed in the standard float32 .”

“Other techniques to accelerate convolutions involve the use of quantization and reduced precision data-types [3, 17], which reduce the tensor memory footprint at the cost of model accuracy.”

“Note that the LHS/RHS matrices are stored as 16-bit data types (2-Bytes) while the accumulation happens in 32-bits (4-Bytes) for output derivation [48].”

“To accelerate the training process of reward model, we employ FlashAttention [27, 28], Deep-Speed [29, 30], and mixed-precision training [31, 32].”

“We use bf16 since it has been shown to achieve the same performance and stability as fp32 [40], and it maintains the same range as fp32.”

“In addition to the presented results, we also introduced bit flips into 16-bit Brain-Float quantised models [42].”

“…both weights storage and computation can be conducted in a lower precision, for example, using 16-bit, 8-bit, or 4-bit FP and Int (fp16, bf16 (Kalamkar et al., 2019), fp8-e4m3, fp8-e5m2 (Micikevicius et al., 2023), int8 (Dettmers et al., 2022), fp4, and int4 (Dettmers et al., 2024)) tensors…”

“To conduct efficient training, each step undergoes a single epoch of training using 8-bit quantization and bfloat16 data format (Kalamkar et al., 2019) for every backbone multimodal LLM.”

“5, comparing PAIR embeddings with embeddings from their corresponding baseline.”

“The experimental studies [7], [8], [17] make us think that DNNs can nevertheless maintain high inference accuracy even with low-precision FP arithmetic.”

“Several research groups have explored the efficacy of floats and BFP on the performance of DNNs with multiple image classification tasks [10], [22]–[25], [32].”

“As demonstrated by Kalamkar et al. [110], BF16 offers a dynamic range equivalent to FP32, eliminating the need for hyperparameter tuning and simplifying precision conversion.”

“Through extensive performance testing, we found that using bfloat 16 (Kalamkar et al., 2019) alongside FlashAttention 2 (Dao, 2023) and data packing provided the most efficient compute performance in our setup.”

“…in OPT 6.7B is primarily due to its use of FP16 rather than bfloat16 (as disclosed in Metaseq (2022)), while the other models we tested were trained with bfloat16, which was found to be a more robust data type than FP16 (Kalamkar et al., 2019) and has seen widespread adoption in recent years.”

“Beyond FP8 formats, alternative formats for DNN training, such as BFloat16 [12], [13] and block floating-point formats [14], have also been investigated.”

“With only minimal modifications, DNNs have been shown to run without accuracy loss using reduced bitwidth floating-point formats such as IEEE’s half-precision (float16), Google’s “brain floating point” (bfloat16), and Nvidia’s 18-bit TensorFloat format—all of which have been implemented on a variety of platforms from…”

“on Ncore has become straightforward and avoids accuracy loss [14].”

“To mitigate this issue, modern FP8 training recipes utilise various scaling techniques before casting from higher-precision formats—typically BF16 [14] for activations—to FP8 formats used in matrix multiplications.”

“[Micikevicius et al., 2018], BF16 [Kalamkar et al., 2019], FP8 [Wang et al., 2018, Sun et al., 2019] and so on.”

“To operate on a consumer GPU, we truncate the maximum sequence length to 256 and use bfloat16 precision (Kalamkar et al., 2019).”

“To ensure training stability, we pretrain all models using bfloat16 (Wang & Kanwar, 2019) (Kalamkar et al., 2019) (Scao et al., 2022), a data type that shares the same dynamic range as float32.”

“In contrast, the average throughput loss for BF16 to MX9 is 1.1% (not shown).”

“Consequently, the performance of LLM training on modern hardware is significantly hindered by the inadequate bandwidth, as the amount of data movements to and from DRAM is the major performance bottleneck.”

“Many mantissa bits in highprecision formats contain little to no information in applications with large uncertainties, such as deep learning [6], [7] or”

“All values are in bfloat16 format [21], a data type commonly used in machine learning areas, which preserves the approximate dynamic range of float32.”

“Prior work in this space has typically required isolated (i.e., proprietary) simulation of number systems within different environments (e.g., bfloat [18], efloat [4]).”

“Current quantization methods have adopted BFLOAT16 [21], INT8, INT4, 2-bit ternary, and 1-bit binary numbers as well as mixed-precision schemes to represent the weights and activations in CNNs. Floating-point numbers and integers-based networks still use multiplication accumulation for the dot product in convolution l…”

“Bfloat16 ’s dynamic range is similar to a 32-bit float but has lower memory traffic and footprint, which makes it appealing for neural networks [Kalamkar et al. 2019].”

“Yu et al. [100] perform analysis by using two 8-bit floating-point quantization techniques and Posit8 and show comparable performance with BFloat16 [101] accuracy while significantly reducing the area and power consumption in hardware implementations.”

“This format was later named Brain Floating Point 16 (BF16) [11] and deployed on Tensor Processing Unit (TPU) [12] V2 and V3 .”

“Similarly, to curtail the cumulative rounding error in deep learning algorithms, low-precision multiplications such as TF32 [9], HP [7], and BF16 [10] must be accumulated into SP format in general matrix multiplication (GEMM).”

“Left: C OLLAGE uses a strict low-precision floating-point (such as BF16) optimization loop without ever needing to upcast to FP32 like in the mixed-precision with master weights (red thick loop).”

“For example, PyTorch has a built-in mixed-precision mode called automatic mixed precision (AMP), which places all operations at float16 rather than float32 , with the exception of reduction operations, weight updates, normalization operations, and specialized operations (Paszke et al., 2019). bfloat16 and TF32 are com…”

“In an effort to further improve hardware effi-ciency, there has been increasing interest in using data types with even lower precision than FP32 for training (Micikevicius et al., 2018; Kuchaiev et al., 2018; Wang et al., 2018; Kalamkar et al., 2019; Mellempudi et al., 2019; Sakr et al., 2019).”

“Further advancements in precision reduction have led to the introduction of Brain Floating Point (BF16) [125], a novel half-precision format.”

“Formats like FP16 [8], BF16 [7], FP8 [6], [9], and INT8 [5] have demonstrated notable success, but it remains uncertain whether precision can be further reduced to 4 bits without compromising convergence or model quality.”

“The bfloat16 format has been extensively employed in the deep learning field for many years and has developed into a well-established standard for both training and inference [59].”

“However, recent results show that half-precision (16-bit) floating point formats provide similar levels of performance as FP32 on the same architecture, while requiring much less memory and decreasing computation time [27].”

“A second problem with using hyperbolic points as intermediate features is that these points can stray far from the origin (just as Euclidean DNNs require high dynamic range Kalamkar et al. (2019)), especially for deeper networks.”

“It has been well known that the accuracy of DNN is sensitive to an exponent rather than a mantissa in a floating-point number system [14].”

“This can significantly reduce computational cost, while maintaining performance, as demonstrated for bfloat16 [ 61 ] , float8 [ 62 ] , and float4 [ 63 ] .”

“Compute cores are coupled to a 64-bit wide SIMD-capable floating-point unit (FPU) supporting FP64 , FP32 , FP16 , Brain-Float16 ( BF16 ) [17], FP8 (E5M2), and FP8ALT (E4M3) [18] formats.”

“A particular variant π op 0 , C of π op , C has received much attention as well (Kalamkar et al., 2019), which assigns low-precision formats to (most of) the input and output tensors of GEMM operators: it is defined as We note that the precision assignments used in apex.amp and torch.amp (Nvidia,…”

“The current trend in low-precision training with hardware support focuses on 16-bit floating-point formats with half-precision (FP16) [3] and bfloat16 [5] alternatives from NVIDIA and Google.”

“HW4∼6 are energy-efficient training accelerators with different BN modules using bfloat16, which is based on the recent studies possibly enabling on-device training [3], [17], [31].”

“Intel’s Bfloat16 [Kalamkar et al. 2019] and FlexPoint [Köster et al. 2017], Nvidia’s TensorFloat32 [NVIDIA 2020], Microsoft’s MSFP [Rouhani et al. 2020], Facebook’s Log Number System [Johnson 2018], and Posits [Gustafson and Yonemoto 2017] are examples of such recent FP variants.”

“One-shot methods (Wanda, SparseGPT) are the cheapest (0.16/0.25 hours with single GPU) but collapse at this sparsity level, yielding unusable perplexities ( ě 10 4 ).”

“For example, 16-bit precision (FP16 [35], BF16 [7, 40]) formats have become a de-facto standard for training and fine-tuning to balance between accuracy and resource consumption.”

“For instance, NVIDIA has supported FP16 and BF16 formats [19] in their A100 GPUs.”

“Note that this keeps the dynamic range of the IEEE 32-bit numbers [22] and has been shown to be beneficial when training neural-networks using 16-bit number representations [23].”

“Recently, DNN training with 16-bit (BFloat16) and 12-bit (TinyFloat12) formats [15, 16] also achieved similar accuracy as 32-bit.”

“On the FP arithmetic side, 16-bit formats such as half precision FP16 (5-bit exponent) and bfloat16 (8-bit exponent) have found success in both inference acceleration and mixed-precision DNN training [19].”

“Pruning neural networks after training [3, 9, 36] and applying specialised data types [5, 28, 59] have the potential of significantly reducing the resource demand with no or only little quality loss.”

“It has been observed that using the bfloat16 floating point format for training convolutional neural networks can match or even exceed performance of networks trained using traditional single precision formats such as fp32 [10, 12, 14], possibly due to a regularizing effect from the lower precision.”

“For example, in processors that support it, computations in BrainFloat16 (BF16) (Kalamkar et al., 2019) can have double the maximum throughput of single precision FP32.”

“The models use mixed precision (Micikevicius et al., 2017) in BF16 (Kalamkar et al., 2019), with gradient AllReduce and accumulation in FP32 for numerical stability.”

“While we did not test training models in full precision, using mixed precision is standard in modern language-model training, offering faster training speeds compared to full precision while maintaining greater numerical stability than standard half-precision [51–53].”

“We use mixed precision training (Micikevicius et al., 2018) in BF16 (Kalamkar et al., 2019) with gradient accumulation and gradient all-reduce/reduce-scatter in FP32 for training stability.”

“For all baselines, we employ FlashAttention [11, 12] and mixed-precision training [31, 49], which are commonly used in LLM training.”

“I. INTRODUCTION DURING the past decade, Deep Neural Networks (DNNs)have shown outstanding performance in a myriad of Artificial Intelligence (AI) applications.”

“Referencing, [24– 26] this paper assumes that the multiplication result is represented as an FP32 value.”

“This is the case of bfloat16 (Brain Floating Point), developed by Google [61] for its TPU architectures.”

“Many mixed-precision approaches have been introduced in recent years [29], [30], [31], [36].”

“The training process is performed in bfloat16 mixed precision (Kalamkar et al., 2019) to enhance training efficiency and avoid numerical underflow or overflow problems.”

“Recent researches have found that computation in low-precision data can maintain the accuracy of deep learning algorithms, while significantly improve the performance[4–6].”

“As illustrated in Figure 2, the final evaluation loss for 4-bit training increases significantly with larger learning rates, whereas BF16 training exhibits a more stable performance across different learning rates.”

“Inference performance was bench-marked on NVIDIA A100 GPUs using ESM2 models ranging from 8 million to 15 billion parameters, with a batch size of 16 proteins in brain floating point (bfloat16) 49 for both original and efficient implementations.”

“We employ bfloat16(Kalamkar et al., 2019) mixed precision training to improve memory efficiency and computational speed.”

“(3) operational or contextual flags, such as session reuse indicators (key-value cache usage) or quantization configurations, revealing sensitive model runtime states or deployment settings.”