A Study of BFLOAT16 for Deep Learning Training

“When training ResNet-50 on Imagenet using the AVX512BF16 instructions, we achieved a Top-1 accuracy of 75.62%, which matches the current state-of-the-art performance.”

“The network uses a “VGG loss” function using a pre-trained VGG19 network [33].”

“Here we choose AlexNet (ILSVRC 2012) [22] and ResNet-50 (ILSVRC 2015) [18] as representative models for the BFLOAT16 evaluation.”

“However, all the aforementioned methods are shown to work for smaller benchmark model/data-sets only and invariably result in a non-trivial drop in accuracy with larger ImageNet data-set [9] and classification task [32].”

“However, all the aforementioned methods are shown to work for smaller benchmark model/data-sets only and invariably result in a non-trivial drop in accuracy with larger ImageNet data-set [9] and classification task [32].”

“Using multiple deep learning frameworks modified to insert appropriate Quantlib calls, we provide SOTA results for: AlexNet [22], ResNet-50 [18], DC-GAN [29], SR-GAN [24], DeepSpeech2 [4], GNMT [37], and two industrial workloads namely: a Deep and Cross Network, and a DNN Recommendation System.”

“We have implemented DC-GAN in Caffe and for our experiments all the input tensors (activations, weights) are converted to BFLOAT16 for convolution layers (in both the generator and the discriminator), while only the input activations are converted to BFLOAT16 for batch normalization layers; all other tensors are maint…”

“There are at least three half-precision formats in the domain of mixed precision training of large neural networks: FP16 [28], 16-bit Integer based [9], and BFLOAT16 [10, 4, 3].”

“The Julia language [5], which is designed to provide high-performance without sacrificing ease of programming, has also been shown to be benefited from BFLOAT16 [11].”

“The BF16 half-precision floating-point format [12] was developed by Google Brain, which is an AI research group of Google.”

“It has been demonstrated that the 32-bit accumulator is a low-overhead, critical feature to handle AI workloads [11], by avoiding absorption (swamping effect) In this article, we are interested, regarding Binary32 and Bfloat16 computation for the summation problem, in: • Predicting a result’s…”

“We can run some layers in a DNN with a much lower data precision than floating-point without losing much accuracy [7, 15], and this reduction in precision results in less computation budget.”

“We compare our approach against BF16 , COAT* (Xi et al., 2024)(COAT with MXFP4 adaptation, excluding optimizer quantization), and the search-based Four Over Six (Cook et al., 2025) base-line(with MXFP4 adaptation).”

“…as GPT (Radford et al., 2019), OLMo (Groeneveld et al., 2024), and the LLaMA family (Touvron et al., 2023) have traditionally relied on mixed precision (FP32/FP16 or FP32/BF16) (Micikevicius et al., 2018) and half-precision (FP16/BF16) (Kalamkar et al., 2019) to optimize computational efficiency.”

“Thefirststrategyinvolvedquantizingmodel parametersandadoptingthehalf-precisionbrainfloating-point format, bfloat16 , designed by Google Brain [48], [49].”

“However, it comes with the cost of the smaller range of representable values [12].”

“The bfloat 16 numeric type [51] is used instead of the standard float32 both in training and tests.”

“To mitigate this burden, various algorithmic techniques for data quantization on activations and/or weights have been explored [5], [8], [14], [21], [25], [31], [34], [35], [54], [60].”

“This high correlation is expected, as the Helpfulness aspect acts as a collective measurement for the other aspects.”

“Recently, BF16 has gained popularity for large models due to its wider exponent range, which offers training stability [32], [35], [54].”

“…systems (Vaswani et al., 2017) are routinely executed in reduced precision to increase throughput and reduce memory cost (Micikevicius et al., 2018; Kalamkar et al., 2019; Dettmers et al., 2022; Yao et al., 2022; Frantar et al., 2022; Xiao et al., 2023; Lin et al., 2024; Bondarenko et al., 2023).”

“This happens because neural network weights normally cluster narrowly around zero, which has led to the use of truncated formats like BFLOAT16 [3].”

“The online kernel itself differs from an exact FP32 reference by 6 .”

“(TPUv2) [17] introduced the brain floating point format , also known as bfloat16 [20].”

“BF16 has seven stored fraction bits and an implicit leading bit, giving eight bits of significand precision (Kalamkar et al., 2019).”

“All stages use BF16 arithmetic (Kalamkar et al., 2019) and the AdamW optimizer with β 1 = 0.9 , β 2 = 0.95 , ϵ = 10 − 8 , and a weight decay of 0.01.”

“R ELATED WORK Mixed-precision training keeps fp32 master weights because fp16 weights alone lost small updates and accuracy [11]; with the copy, bf16 matches fp32 [25].”

“Our primary experiments use FLUX.1-Dev 1 in bfloat16 precision [19] on a single NVIDIA A6000 GPU (48GB VRAM), with 21 denoising steps and a guidance scale of 4.5.”

“BF16 tensors are their round-to-nearest-even conversions, which approximates BF16 mixed-precision training [33].”

“Trainable weights, gradients, Adam (Kingma and Ba, 2015) states, and all readout operations use FP32, while the retained language layers compute in BF16, following standard mixed-precision training (Mi-cikevicius et al., 2018; Kalamkar et al., 2019).”

“BF16 / FP16 remain common for training-state updates and optimizer paths [3], [4], FP8 variants (E4M3/E5M2) target high-throughput training and serving [5], [6], FP4 is emerging for aggressive inference [7], and Micro Scaling (MX) formats pair low-precision elements with shared block-scale…”

“In contemporary machine learning frameworks, mixed-precision execution has become standard practice [1], [2], [3], [4].”

“Training uses bf16 mixed precision [88, 89] on an H20 cluster with a global batch size of 128.”

“For efficient large-model training, we use FSDP2 [34], together with bfloat16 precision [35], fused kernels, and gradient checkpointing [36].”

“We train each model on eight NVIDIA A100-SXM4-80GB GPUs using distributed data parallelism (DDP) (Li et al., 2020) with bfloat16 precision (Kalamkar et al., 2019).”

“Training is distributed over 4 GPUs with bfloat16 mixed precision [77, 78].”

“Empirical results reported in prior work [35] demonstrate that training with BF16 can be achieved without degradation in convergence or accuracy, even for large-scale models.”

“Mixed-precision floating-point arithmetic is a fundamental driver of scale in modern deep learning (Gupta et al., 2015; Micikevicius et al., 2018; Kalamkar et al., 2019; Micikevicius et al., 2022).”

“We train with BF16 mixed precision [27, 41] and apply activation checkpointing at the input to every layer [7].”

“Precision formats and outlier activations. fp16 has 5 exponent and 10 mantissa bits (maximum 65,504); bf16 trades precision for fp32 ’s dynamic range with 8 exponent and 7 mantissa bits (Kalamkar et al., 2019).”

“All experiments are based on the Voxtral-mini-3B model [35], which is based on the Ministral-3B model [51], and uses brain floating point (bfloat16) precision [52].”

“The core framework comprises head personalization, bfloat16 Kalamkar et al. [2019] quantized communication, and a tunable active-set cardinality κ .”

“For example, many approaches utilize quantization that exploits data types with a low bit width such as half-precision (FP16) and BFloat16 (BF16) [7], FP8 [8], [9], dynamic floating-point [10], block floating-point [11], [12], flex-point [13], or fixed-point [10].”

“BF16+TF32 mixed precision [Kalamkar et al. 2019] and gradient checkpointing [Chen et al. 2016] kept memory within the bounds of a single NVIDIA RTX A6000 (48 GB VRAM, 8-core CPU, 45 GB RAM).”

“Training used bfloat16 [32] mixed precision, a physical and effective batch size of 32 cells, and no gradient accumulation.”

“The format definitions are as follows [37]–[39], [41]: TF32 was excluded because it is a compute mode and not a storage type [40].”

“When the original and flipped values are both finite, flipping e j changes the magnitude by [Kalamkar et al., 2019] | flip e j ( g ) | | g | = 2 2 j , e j = 0 , 2 − 2 j , e j = 1 .”

“These advantages motivated the adoption of 16-bit floating-point formats, such as Google’s Brain Floating Point 16-bit (BF16) [14], as a standard alternative to the full-precision floating-point format (FP32) in the training of early large language models (LLMs).”

“…8 × 8 , dims to 64 via multi-pass with FP32 partial-sum round-trips through the output SRAM) but is training-specific: three dataflows including accumulate-onto-memory dW , an FP32 psum path with a fused bias/activation/cast drain epilogue, and BF16-in/FP32-accumulate numerics [7], [6] throughout.”

“Formats such as BFP16 [13] and FP8 [14] further reduce activation memory footprint without requiring calibration, but these floating-point formats still remain computationally complex.”

“The optimisation recipe (AdamW, cosine warmup, RMSNorm, EMA, bf16) follows standard practice [33,32,55,25] and numerical stability findings [27].”

“A common way to accelerate linear layers is low-precision computation (Kalamkar et al., 2019; Lee et al., 2009; Micikevicius et al., 2017; Liu et al., 2024).”

“…is complemented by standard stability and efficiency utilities, including cosine (Loshchilov and Hutter, 2016) and linear learning-rate schedules with warm-up (Goyal et al., 2017), gradient accumulation, gradient clipping (Pascanu et al., 2013), and bfloat16 mixed precision (Kalamkar et al., 2019).”

“The relative benefits of AMP are expected to differ on Apple Silicon M2 Pro (which lacks dedicated Tensor Cores but has a unified memory architecture) and on cloud A100 GPUs (which support BFloat16 as an alternative to FP16 [9]).”

“To maximally maintain the accuracy of LT models without requiring additional fine-tuning, we select two representative low-precision formats that are widely adopted in DNN model implementations: the Brain Floating Point (BF16) [11] and the Micro-scaling INT8 (MXINT8) [24] formats.”

“This model with a size of 312 KB using FP16 [33 ] is ideal for deployment on desk-top class processors e.g. quad-core Intel processors like the Intel Core i3 [34] or Intel Core2 Quad [35 ], as their substantial memory capacities (e.g., up to 64 GB DDR4) and large cache sizes (e.g., 8 MB L2 for…”

“Kalamkar et al. [23] showed that BF16’s wider range can eliminate loss scaling in most training runs.”

“This section presents inference-accuracy results comparing Posit, IEEE 754 FP, and Google’s BF 16 [13].”

“Unless otherwise stated, experiments use GPT-2 Medium [22] trained from scratch on OpenWebText with sequence length 1024, AdamW [17], BF16 precision [13], and a fixed token budget of 49.8B tokens.”

“…counts of foundational models scale into the billions, the adoption of resource-efficient, low-precision arithmetic formats, such as BF16, FP8, and emerging 4-bit structures like MXFP4, has become critical [Kalamkar et al., 2019, Micikevicius et al., 2022, Mishra et al., 2025, Tseng et al., 2025].”

“By adopting lower-precision formats, such as tensor floating point (TF32) [7], half-precision (FP16), brain floating-point (BF16) [8], and float8 (FP8) [9], these quantization techniques effectively reduces the bit-width of data representations.”

“While traditional IEEE-754-like 16-bit number formats such as Float-16, BFloat-16, and TensorFloat-16 [8]–[10] have been successfully employed for such inference, comparable accuracies at even lower bit-widths have been demonstrated [11]–[13] using the novel Posit number-format [14].”

“For example, the inference phase for large language models (LLMs) often employs a 4-bit integer and 16-bit floating-point (INT4 × FP16) format to maintain accuracy [14], while the training phase relies on high-precision floating-point MAC operations, such as brain-float 16 (BF16) or FP32 [15].”

“Training uses bfloat16 for the model copy used in the forward and backward passes [22].”

“To optimize throughput and memory utilization without compromising numerical stability, we employed Bfloat16 (BF16) mixed-precision training [28].”

“Both formats halve memory footprint relative to fp32 and are natively supported on modern GPUs and TPUs [6, 7].”

“To mitigate these costs, the bfloat16 (BF16) [24] format has been widely adopted in recent PIM-based accelerators, as it preserves the 8-bit exponent of FP32 while reducing the mantissa to 7 bits, maintaining inference accuracy with substantially lower implementation cost.”

“The bf16 mixed-precision strategy [22] was used to improve efficiency and reduce GPU memory usage.”

“A variety of fault-tolerance techniques have been explored in prior works for edge NPUs using conventional data formats, e.g., INT8/16 [33, 48], FP16 [15, 42], BF16 [15, 35].”

“Mixed-precision training is well established in deep learning [54], [55].”

“Crucially, BF16 maintains the same exponent range as full-precision FP32 while reducing memory and computational overhead [1] [10].”

“27 We convert the stored values back to the larger bit-widths of either single or double precision during time-reversed simulation.”

“BF16[16] subsequently became widely adopted due to its larger exponent range relative to FP16, offering improved numerical stability with comparable storage cost.”

“We utilized BFloat16 mixed-precision training to optimize memory usage and stability (Kalamkar et al., 2019).”

“…commonly uses mixed precision : many tensors are kept in low precision (e.g., FP16/BF16/TF32) for throughput, while critical accumulations (e.g., master weights and optimizer updates) are performed in FP32; in FP16, (dynamic) loss scaling is used to mitigate gradient underflow [41, 36, 43].”

“Lower-precision arithmetic operations, such as 16-bit floating-point (FP16) and brain-float (BF16) formats, execute significantly faster than their full-precision counterparts while maintaining precision sufficient for algorithm [6, 5].”

“Modern AI accelerators, such as tensor processing units (TPUs) and neural processing units (NPUs), are predominantly optimized for low-precision arithmetic (FP16/BF16) [6 , 14] to maximize throughput and energy efficiency.”

“BLEU scores are in the appendix Figure 4.”

“To mitigate these burdens, recent research has increasingly turned to more efficient numerical formats—initially FP16 [27] and BFloat16 (BF16) [17], and more recently low-bit quantization, particularly 8-bit and 4-bit.”

“The existing method can also be modified to accommodate AI-specific floating point formats such as bfloat16 [22] and DLFloat [23].”

“The introduction of BF16 format addressed some numerical stability issues of FP16 by providing a wider exponent range at the cost of mantissa precision [7].”

“BFLOAT16, in particular, has been widely adopted for its compatibility with existing hardware and its ability to balance precision with computational efficiency [3].”

“2 Low-bit FP formats such as BrainFloat [20] and Ten-sorFloat [22], adopted by Google and NVIDIA respectively, reduce the bit-width of mantissas while retaining 8-bit exponents (e.g., 1s7m8e and 1s10m8e).”

“Beforeexecution,AMXtilesmustbe configuredusingthe LDTILECFG instruction to specify the number of rows (up to 16), bytes per row (number of columns), and starting row [65].”

“Additionally, emerging numerical formats, such as TensorFloat-32 (TF32) and Brain Floating Point Format (BF16), offer balances be-tween computational speed and numerical accuracy [9], [10], providing more flexible options for quantization.”

“BFLOAT16, a 16-bit floating-point representation format proposed by Google (Kalamkar et al. 2019), is a truncated version of FP32.”

“Another important consideration is native support for the BF16 (bfloat16) tensor data type 26 .”

“…the fact that many data-intensive applications like big data analytics, deep learning, and image processing can tolerate a certain degree of error (Kalamkar et al. 2019; Abdelfattah et al. 2020), there exists a wide range of scientific and numerical applications, including weather forecasting (He…”

“FP16 and FP32 are the standard half-precision and single-precision FP formats, BF16 is Google’s bfloat16 format [7], and TF32 is NVIDIA’s TensorFloat-32 format [12].”

“Common floating point types include IEEE single precision, FP32 mode for single precision, IEEE half precision [8], and bfloat16 [1], [7].”

“One widely adopted format is bfloat16 Kalamkar et al. (2019), which retains a wider range of values compared to traditional FP16 while providing faster computation and reduced memory consumption.”

“In a notable contribution, the Google team introduced the Brain Floating Point format [8].”