A Study of BFLOAT16 for Deep Learning Training

“When training ResNet-50 on Imagenet using the AVX512BF16 instructions, we achieved a Top-1 accuracy of 75.62%, which matches the current state-of-the-art performance.”

“The network uses a “VGG loss” function using a pre-trained VGG19 network [33].”

“Here we choose AlexNet (ILSVRC 2012) [22] and ResNet-50 (ILSVRC 2015) [18] as representative models for the BFLOAT16 evaluation.”

“However, all the aforementioned methods are shown to work for smaller benchmark model/data-sets only and invariably result in a non-trivial drop in accuracy with larger ImageNet data-set [9] and classification task [32].”

“However, all the aforementioned methods are shown to work for smaller benchmark model/data-sets only and invariably result in a non-trivial drop in accuracy with larger ImageNet data-set [9] and classification task [32].”

“Using multiple deep learning frameworks modified to insert appropriate Quantlib calls, we provide SOTA results for: AlexNet [22], ResNet-50 [18], DC-GAN [29], SR-GAN [24], DeepSpeech2 [4], GNMT [37], and two industrial workloads namely: a Deep and Cross Network, and a DNN Recommendation System.”

“We have implemented DC-GAN in Caffe and for our experiments all the input tensors (activations, weights) are converted to BFLOAT16 for convolution layers (in both the generator and the discriminator), while only the input activations are converted to BFLOAT16 for batch normalization layers; all other tensors are maint…”

“There are at least three half-precision formats in the domain of mixed precision training of large neural networks: FP16 [28], 16-bit Integer based [9], and BFLOAT16 [10, 4, 3].”

“The Julia language [5], which is designed to provide high-performance without sacrificing ease of programming, has also been shown to be benefited from BFLOAT16 [11].”

“Bfloat16 (BF16), which was introduced by Google [3], has the same range as FP32 at the expense of mantissa bits.”

“This improvement not only enhances computation speed but also accelerates model convergence [77].”

“Studies showed that BF16 can achieve similar results as FP32 while having significant speedup and memory savings (Kalamkar et al. 2019).”

“While this can be as simple as reduced-precision floating point [18], more radical approaches include stochastic computing [45, 46] and race logic [25], which experiment with alternative data encodings.”

“Brain floating-point (BFloat16) is proposed to be applied in neural networks to replace 32-bit single-precision floating-point numbers [66].”

“Similarly, new data formats with a reduced bit-width have been proposed, for example, BFLOAT [4] and POSIT [5] can be used both during training and inference time.”

“The bfloat16 (a.k.a. Brain Floating Point) floating-point format [2] developed by Google Brain is basically a truncated (16-bit) version of FP32, which is broadly used for low-cost training and inference of deep learning [3].”

“As baselines, we use the FP16b (brain FP format), used in modern processing units to accelerate NN training [4] and are used in the state-of-the-art for encoding Bop momentum values for efficient BNN training in [11], and FP32.”

“The ’Brain’ floating-point format (bfloat16) was developed to mitigate this issue [26].”

“In addition, Google proposed another 16-bit format, named Brain Floating-Point Format (BFP16), simply by truncating the lower half of the FP32 format [10].”

“BF16 has the same accuracy as FP32 for NN training [47] but is more cost-efficient.”

“It’s noteworthy that in our experimental setup, we employ BFloat16 (Kalamkar et al., 2019) due to its observed advantage in enhancing the training stability of TransNormerLLM models.”

“To further decrease the computational demands necessary for LLM fine-tuning and inference, MPFT via mixed-precision (i.e., FP16 or BF16 ) [27, 38] and quantized low-precision [12] have proven effective strategies to reduce GPU memory and runtime requirements without deleterious effects on…”

“Specifically, the smallest singular value for bfloat16 [Kalamkar et al., 2019] can be on the order of 10 − 12 in some cases.”

“…section, optimization is used with AdamW (Loshchilov, 2017) and a cosine-annealed learning-rate schedule with warm-up steps (Loshchilov & Hutter, 2016 or official evaluation scripts for its VQ-VAE; we therefore reproduced its evaluation using the publicly released weights and their GitHub codebase.”

“We use bfloat16 (BF16) precision for training to optimize memory efficiency and throughput with minimal impact on accuracy (Kalamkar et al., 2019).”

“18 × 10 − 38 ) has made BF16 widely adopted in both LLM training (Kalamkar et al., 2019) and inference (Kurtic et al., 2025).”

“…the computational and memory footprint of LLMs by representing model weights and activations with low-precision data types, such as INT8 (Dettmers et al., 2022), FP4, or NF4 (Dettmers et al., 2023), instead of high-precision floating-point formats like FP32, FP16, or BF16 (Kalamkar et al., 2019).”

“All training and evaluations are performed using brain floating point half-precision (BF16) [22] to reduce memory footprint and accelerate training and inference speed.”

“Training is performed with a global batch size of 384 on four 48G GPUs used for all experiments, using BF16 mixed precision [30].”

“For Core Attention, the thread blocks are split among the different KV heads, and for each KV head, each SM retains a replicated copy of queries which multiply with that particular KV head.”

“…attention mechanism of T5 and compute all operations including separate multiplications and function evaluation using single-precision and BFloat16 [29] floating-point arithmetic; (b) Models ’FP32-ExpMul’ and ’BF16-ExpMul’ employ floating-point arithmetic for all parts of the model except for the…”

“This is analogous to the introduction of the ‘BFloat16’ [18] format, which achieves superiority over the older IEEE standard 754 ‘Float16’ [17] in certain deep machine algorithms [22] by allocating 8-bits towards the exponent, as the larger ‘Float32’ format does.”

“BF16 and FP32 are shown to result in a similar cross-entropy loss at each step, so BF16 can provide sufficient trainability, which is aligned with the statement by Kalamkar et al. [22].”

“Therefore, bfloat16 is often preferred for its increased dynamic range (Kalamkar et al., 2019; Almazrouei et al., 2023) and block scaled data formats have been developed to increase dynamic range while maintaining low precision (Rouhani et al., 2023).”

“Benefiting from BF16 [29], our model can be trained with a total batch size of 112.”

“Meanwhile, Google introduced an efficient representation referred to as Brain float 16, which follows the same structure but utilizes different sizes for the exponent and mantissa [21].”

“To train the model stably, we conduct the following tricks: • we use BF16 (Kalamkar et al., 2019) instead of FP16.”

“The use of different number types in machine learning applications has been analyzed extensively in previous years, more specifically in deep learning neural networks [1,2].”

“If it is successful we would also obtain the merits of both ours and BFLOAT16’s. Methodologically, we did almost the same as [8], i.e. pseudo multiplications were done for and resulted in BFLOAT16, then they are accumulated in FP32, and weights were kept in FP32.”

“BFloat16 [10] was introduced for deep learning applications [11] and has the same exponent size as the FP32.”

“This improvement not only enhances computation speed but also accelerates model convergence [77].”

“The industry widely quantizes both the model weights and the input values to low-bitwidth numbers such as BFLOAT16 [3], INT8 [4], [5], [6], FP8 [7], FP6, and even INT4 [8], [9] and FP4 [10], [11] on central processing unit (CPU) and graphic processing unit (GPU) to reduce memory usage,…”

“Kalamkar et al. (2019) showed that BFloat16 (Google, 2019) works equally well, and Za-mirai et al. (2020) explored pure BF16 master weights with stochastic rounding and Kahan summation.”

“As a result, BF16 is widely adopted in modern accelerators for both training and inference [28].”

“Key features of this method include: —Extended accumulator width to prevent precision loss.”

“To improve efficiency and reduce memory consumption, we employ FlashAttention-2 [41], ZeRO [42], and BFloat16 mixed-precision training [43], [44].”

“Furthermore, we aim to explore adaptive mechanisms that dynamically address model heterogeneity, making CoM more versatile in real-world deployment scenarios.”

“The designs exploit input sparsity to achieve significant reductions in compute cycles [15-17].”

“…specific NSs’ suitability for DNNs, adapted existing NSs to suit DNN workloads, or proposed new NSs tailored specifically for DNNs. Leading companies such as Google [15], NVIDIA [16], Microsoft [13], IBM [17], and Intel [18], [19], [20] have contributed significantly to this field’s advancement.”

“Due to limited computational resources, models with more than 1.5 billion parameters were loaded in half-precision using bf16 (Kalamkar et al., 2019) to mimic consumer-grade technology limitations.”

“We trained all our models with AdamW [36], using mixed precision: our model parameters, gradient accumulation and communication buffers are in bfloat16 [24] while our sharded optimizer states are in single precision (see Fig.”

“To reduce GPU memory usage and increase the pace of training, all experiments - excluding LLaMA-2-7B-ARK-FP32 were carried out with BFLOAT16[Kalamkar et al., 2019] half-precision format.”

“Several works already explore mixed precision to improve throughput, latency and energy to performance ratio [65], [66], [67], [68], [69], [70].”

“Each experiment is computed using different formats for floating-point—32-bit single precision, 16-bit half precision, and bfloat16 format (Kalamkar et al. 2019)—, and posit—from 32 down to 8 bits length—number formats.”

“To counter the loss in precision, applications often need to use a mixture of number formats, for instance in numeric linear algebra [8, 27, 45] and in deep learning [37, 54].”

“Results in the literature have shown the potential of using bfloat16 format in training machine learning algorithms [22].”

“For example, BFloat16 [10] reduced the mantissa down to 7-bits, and HFP8 [7] lowered the bit-precision to ( 1 - 4 - 3 / 11 ) for the inference computation to gain 2 - 4 × hardware efficiency.”

“For 16-bit encodings ( N = 2 16 ) , we use bfloat16, a standard which is commonly used in machine learning [6].”

“A previous work [160] demonstrates, training with BFLOAT16 can achieve the same accuracy as training with FP32.”

“Moreover, state-of-the-art CPUs have several features beneficial for neural networks, such as the vectorized fused multiply-add (FMA) instruction [21] and the half-precision floating point format [24], also referred to as bfloat16 (brain floating point).”

“The multiplier tree in A100 supports more data formats than V100, including BF16 [29], FP16, and TF32.”

“Other representations are actively being researched (Kalamkar et al., 2019; Das et al., 2018; Song et al., 2017).”

“Current 16-bit implementations include IEEE 754 half-precision (FP16), with 1 sign bit, 5 exponent bits, and 10 fraction bits; Brain Floating Point (BFLOAT16) [40, 65], with 1 sign bit, 8 exponent bits, and 7 fraction bits; and Deep Learning Float (DLFloat) [4], with 1 sign bit, 6 exponent bits, and 7 fraction bits.”

“Therefore,Quantizer preserves themost importantmodel parameters in the high precision bfloat16 [52] format.”

“Moreover, we measure reasoning latency on a single NVIDIA A100 GPU with a batch size of 1 , while enabling FlashAttention-2 (Dao, 2023) with BF16 (Kalamkar et al., 2019).”

“The added loop of blocks also allows distributing the computation to more GPU blocks when offloading is used [12].”

“} 1 ), along with improvements in both performance and energy efficiency [6], [8], [15], [16], [25], [35], [42], [43].”

“This question has existed since the introduction of floating-point numbers, but has recently become much more challenging due to the adoption of low-precision data types like BF16 [18] and FP8 [25,33,40] in large-scale training.”

“1 and 3 for various hidden dimension sizes ( d ) and for two reduced-precision floating-point formats: BFloat16 [30] and FP8-E4M3 [31].”

“Training was conducted on 8 H100 GPUs over 4 days with mixed-precision (bf16) (Kalamkar et al., 2019) for speed.”

“BFLOAT16-based multiply-accumulate (MAC) units are adopted in PEs [57].”

“During fine-tuning, to prevent precision loss, the quantized weights are dequantized back to higher precision (16-bit BrainFloat) before being used to update the LoRa adapter.”

“Therefore, we keep the MVU int8 precision and modify the rest of the NPU pipeline to use Google’s 16-bit Brain floating-point precision ( bfloat16 ) which was shown to provide similar inference accuracy to single-precision floating-point precision [35].”

“To mitigate this burden, various algorithmic techniques for data quantization on activations and/or weights have been explored [5], [8], [14], [21], [25], [31], [34], [35], [54], [60].”

“For example, it is only recently that chip manufacturers started rolling out devices that provide support for these custom precisions, among which, the Bfloat16 format, that is a 16-bit format invented by Google for AI applications [7, 10].”

“These are popular formats that are already implemented in hardware and have already been used to reduce precision while maintaining performance and accuracy in different code applications, including in some cases, deep learning algorithms [13].”

“For training, different data formats like 16bit floats (FP16), BFloat [52], DLFloat [53] etc.”

“BFloat16 (Brain Floating Point) [31] is a data representation used in machine learning applications and intended to reduce the storage requirements and increase computing speed without losing precision.”

“First, alternatives such as unums/posits [26, 37], BFloats[38], logarithmic arithmetic [3], and others [29, 43] potentially extract more useful representational power out of the same number of bits, or have range/precision tradeoffs that are more suitable for some modern workloads such as machine learning.”

“The algorithm starts the training process using the SoA MP training [8] for several batches, defined by numBatchesMP parameter.”

“To date, most efforts are focused on approaches that only improve computation efficiency, such as quantization [6], [23], [40], weight pruning [5], [11], input masking [37], temporal redundancy reduction [1], model cascades [28], alternative numerical representations [12], [14], [16], image resolution resizing [31], a…”

“To increase the dynamic range while still using 16 bits only (including the sign bit), the BFloat16 (BF16) [6] format uses 8 bits for the exponent and 7 for the mantissa.”

“Recently, deep learning has led to the emergence of reduced formats such as Microsoft’s msfp8 Minifloats (8 bits), IEEE-754 binary16 (16 bits), bfloat16 (16 bits) [6], or posits (8, 16, 32, and 64 bits) [7].”

“Figure 2b shows the FMA implementation used by Kalamkar et al. in [21], which is similar to the one previously described.”

“One solution is to use low-precision inputs, e.g. BFloat16 (Kalamkar et al., 2019; Dillon et al., 2017), to such functions in order to achieve high-performance activations at the cost of lower accuracy.”

“Measurements are extracted on an isolated compute node with four Tesla A100 80GB and bf16 mixed precision [28].”

“The three candidates for reduced storage formats are FP16, TensorFloat [20], and Bfloat16 [17].”

“Quantization-aware training uses a straight-through estimator: the forward pass uses the codes, and the back-ward pass updates the latents, kept here in fp32. bf16 near the threshold. bf16 has an 8-bit significand (Kalamkar et al., 2019).”

“To optimize training efficiency and alleviate GPU memory overhead, we incorporate bfloat16 (BF16) mixed-precision training [22] along-StructSplat”

“This property is particularly important for the deployment of reasoning-intensive tasks, where lossy techniques can introduce instability and unpredictable accuracy degradation [24], [28], [30], [42].”

“To evaluate the impact of hardware faults across modern mixed-precision regimes, we run all fault-injection campaigns under three training formats: IEEE FP16, BF16 [23], and FP8 [24], which are the de facto low-precision standards.”

“Moreover, the use of low-precision floating-point formats, such as bfloat16, has been adopted in lightweight designs to minimize area and power consumption while maintaining computational accuracy [19].”

“9 × speed-up versus the 32-bit floating point counterpart [31] without sacrificing accuracy [46].”

“Mixed-precision training was enabled using bfloat16 [23].”

“While BF16 is the current de facto standard widely adopted in large-scale model training (Kalamkar et al., 2019), a paradigm shift is underway.”

“Besides, the wider exponent range can even enhance the accuracy of LLM inference with possessing excessive data range [27].”

“Brain Floating Point half-precision format (BF16) [14] and 8-bit floating point (FP8) [15], [16] have been proposed for deep neural network (DNN) training and applied on NVIDIA high-end GPUs.”