A Study of BFLOAT16 for Deep Learning Training

“When training ResNet-50 on Imagenet using the AVX512BF16 instructions, we achieved a Top-1 accuracy of 75.62%, which matches the current state-of-the-art performance.”

“The network uses a “VGG loss” function using a pre-trained VGG19 network [33].”

“Here we choose AlexNet (ILSVRC 2012) [22] and ResNet-50 (ILSVRC 2015) [18] as representative models for the BFLOAT16 evaluation.”

“However, all the aforementioned methods are shown to work for smaller benchmark model/data-sets only and invariably result in a non-trivial drop in accuracy with larger ImageNet data-set [9] and classification task [32].”

“However, all the aforementioned methods are shown to work for smaller benchmark model/data-sets only and invariably result in a non-trivial drop in accuracy with larger ImageNet data-set [9] and classification task [32].”

“Using multiple deep learning frameworks modified to insert appropriate Quantlib calls, we provide SOTA results for: AlexNet [22], ResNet-50 [18], DC-GAN [29], SR-GAN [24], DeepSpeech2 [4], GNMT [37], and two industrial workloads namely: a Deep and Cross Network, and a DNN Recommendation System.”

“We have implemented DC-GAN in Caffe and for our experiments all the input tensors (activations, weights) are converted to BFLOAT16 for convolution layers (in both the generator and the discriminator), while only the input activations are converted to BFLOAT16 for batch normalization layers; all other tensors are maint…”

“There are at least three half-precision formats in the domain of mixed precision training of large neural networks: FP16 [28], 16-bit Integer based [9], and BFLOAT16 [10, 4, 3].”

“The Julia language [5], which is designed to provide high-performance without sacrificing ease of programming, has also been shown to be benefited from BFLOAT16 [11].”

“The vector processor includes a vector ALU and a MUL unit for vector integer arithmetic, a vector floating-point unit that operates on Bfloat16 datatypes [24], and a vector permutation unit.”

“For FP, BrainFloat [28] adopts a 1s8e7m format (1 sign bit, 8 exponent bits, and 7 mantissa bits), which preserves the dynamic range of FP32 while reducing mantissa bit-width to minimize multiplier overhead.”

“We choose BF16 for implementation because (a) BF16 has a larger dynamic range than FP16 [33], thereby capable of handling extreme values [6], [40], and (b) BF16 is more computationally efficient than FP32 [31], [50], [84].”

“Meanwhile, training throughput on modern GPUs continues to rise via reduced-precision arithmetic (BF16/FP16/FP8) and architectural advances [11, 43, 54, 57, 76], widening the rate gap between ETL and training and leading to idle accelerators and longer time-to-freshness for online models.”

“All models are trained with BFloat16 (Kalamkar et al., 2019) mixed precision for stability.”

“Tatsumi et al. [12] studied the impact of fixed-point and FP formats, multiplication rounding, and FP exceptions FP formats used in the DNN tasks from the literature [30], [31], [32]. on DNN training.”

“While a variety of numerical precisions, such as BFloat16 [20] or INT4 could theoretically be considered for weights and activations to further extend set of possible solutions, the absence of readily accessible hardware platforms that natively support these format necessitated the limitation of…”

“Kalamkar et al. (2019) uses another mixed precision technique with 32-bit floating-point and BFloat, Google Brain’s half-precision format, in which a tensor modification method is used to zero out the lower 16 bits of the 32-bit data flow.”

“Mixed precision strategies, including the use of half-precision formats, have been explored as a way of simultaneously reducing training time and model size while minimizing [24].”

“For instance, low-precision arithmetic has been shown to decrease running time with little to no effect on the ultimate learning task [12, 16], and new floating point formats have been developed and supported in hardware to implement highly efficient low- and mixed-precision computation [17, 34].”

“In subsequent work [7], Intel and Facebook studied the effectiveness of bfloat16 on convolutional and recurrent neural networks, as well as on generative adversarial networks and industrial recommendation systems.”

“• Training using reduced precision arithmetic on 16-bit floating point or even quantized representations [76, 78, 45] significantly reduces memory [27, 50].”

“Both bfloat16 and float16 training [34] are stable and convergent for most models, and LLMs have pioneered many techniques for further lower-precision optimizations [35], [36].”

“…Tang, and Heinecke [22] specifically advocated BF16 for HPC iterative solvers, projecting convergence over “a large range of condition numbers” when an FP32 outer refinement loop is used; Kalamkar et al. [41] clarified BF16 semantics as a 16-bit format with a 7-bit mantissa and FP32 dynamic range.”

“The bfloat16 (BF16) format (Kalamkar et al., 2019) is the standard storage format for KV-cache tensors in current LLM serving systems.”

“During dataset construction, we manually inspected the data to mitigate risks of including sensitive information, offensive language, or potentially harmful stereotypes and biases in the text.”

“In practice, the evaluation of such deep compositions ( inference ) is performed in floating-point (FP) arithmetic, low-precision formats being routinely employed to improve performance and energy efficiency [18, 24, 20].”

“Brain floating-point format (BF16) [23] is applied to accelerate the training and decoding processes.”

“A common compromise is to keep activations in FP16/BF16 while quantizing weights, which improves stability but retains higher memory traffic and power consumption [29, 17].”

“Mixed precision techniques, which employ reduced floating-point precision representations (e.g., bfloat16 [35]) for computations, have been introduced to alleviate memory bandwidth constraints and reduce arithmetic time.”

“Note that since bfloat16 use the same number of bits for the exponent as float32 , the two formats represent the same range of values [12].”

“The BF16 (Google Cloud, 2019) floating-point format was used as it approximates the dynamic range of FP32 with the 50% memory reduction FP16 provides (Kalamkar et al., 2019), which is important to maximise compute resources while maintaining training stability.”

“Another optimization we employ is the use of Brain Floating Point 16-bit (bfloat16) mixed precision for training and validation [22].”

“ANNs have been frequently explored to utilize low-precision data formats (Courbariaux et al., 2015; Kalamkar et al., 2019; Micikevicius et al., 2022).”

“Google utilized the Bfloat16 format with an 8-bit exponent and a 7-bit mantissa [3] in TPU accelerators and achieved equivalent training accuracy to FP32 [29].”

“We utilize Automatic Mixed Precision (AMP) with bf16 format [28] for computational efficiency, and use high precision for matrix multiplications for accuracy.”

“To enhance learning efficiency, we applied bf16 (Kalamkar et al., 2019) and tf32 (Stosic and Micikevicius, 2021) strategy.”

“In fact, mixed precision is a very popular technique actively used in deep learning applications [34, 35].”

“…utilization while maintaining the performance of the training process on it, as given in Table I. BF16 is globally applied in calculations as [5] reported that deep learning training using BF16 tensors achieves the same state-of-the-art results across domains as FP32 tensors in the same…”

“The 32-bit format can be considered the FP 32-bit format, 16-bit as the BF16 format [19], and 8-bit as the hybrid-FP8 [15] format.”

“A previous work [35] demonstrates, training with BFLOAT16 can achieve the same accuracy as training with FP32.”

“Thirdly, drawing from the successful application of low precision in deep learning [25], [26], a relatively low error tolerance ε , specifically 10 − 12 , is deemed sufficient in Q-Newton .”

“…components of each complex number to the following precisions: double precision (float64, 128 bits/complex number), single precision (float32, 64 bits/complex number), half precision (float16, 32 bits/complex number), and bfloat16 (Figure 1), a precision popular in machine learning [18, 19].”

“Finding the appropriate number representation for NNs has already been extensively investigated in the literature, with the evaluation of bfloat16 [12, 13], posits [14, 15, 16], Microsoft floating point [17], FlexPoint [18], or adaptive floating-points [19].”

“Brain floating-point format (BFloat16) [7] is a number encoding format occupying 16 bits.”

“Two major techniques to make DNNs more efficient are (1) reduced-precision Micikevicius et al. (2017); Wang et al. (2018); Sun et al. (2019); Das et al. (2018); Kalamkar et al. (2019); Mellempudi et al. (2020), and (2) sparse representation.”

“To evaluate the impact of Bfloat16 on EAM, we integrate Bfloat16 with the DNNs through PyTorch framework [44].”

“To lower the storage requirements, on decent graphics cards, at least the weight bitmap could be reconfigured to store bfloat16 samples as they are already used for machine learning (Kalamkar et al., 2019).”

“Some are based on emulation [21], while others target existing hardware [28].”

“BFloat16 data type was developed based on an observation that for deep learning models, the reduced precision of a FP32 floating point number does not require additional training epochs — a study from Intel Labs [17] has shown that both the FP32-versions of various deep learning models and their BFloat16 equivalent ca…”

“In [49], float16 (half precision floating point), in [50], bfloat16 (Brain Floating Point), and in [51], DLFloat (Deep Learning Floating Point) data formats have been explored.”

“To further reduce RAM usage, we adopt BF16 (brain floating point with 16 bits) [13] precision as an option to store hidden states throughout the SSM-subgraph.”

“Our training pipeline is based on PyTorch Lightning and we use bfloat16 ((Kalamkar et al., 2019)) and DeepSpeed ((Rasley et al., 2020)) for training optimization.”

“The most direct way to reduce tensor volume is by using datatypes that use fewer bits per value, e.g., BFloat16 [3], half-precision floating-point (FP16), dynamic floating-point [4], flexpoint [5]), or fixed-point [4], [6], [7], [8]).”

“For a nonzero, finite FP32 value x in the normal range of BF16, let e ( x ) = ⌊ log 2 | x |⌋ . e ( x )+1 ] are BF16 has 7 explicit fraction bits (8-bit significand precision) [11], so adjacent values within [2 e ( x ) , 2 spaced by h ( x ) = 2 e ( x ) − 7 .”

“MPT dominates training acceleration pipelines [10], [14], [15], [16], [17], [18].”

“Low-precision arithmetic is what makes large-scale pretraining affordable (Micikevicius et al., 2018; Kalamkar et al., 2019).”

“BF16/FP16 serving is standard (Dettmers et al., 2022; Micikevicius et al., 2018; Kalamkar et al., 2019), but its output-level consequences and targeted mitigation remain incompletely characterised.”

“Finally, we assume models are deployed using bfloat16 (Kalamkar et al., 2019), the standard format for modern model deployments (HuggingFace; Wang & Kanwar, 2019); we test alternatives in §4.4.”

“Bfloat16 transformer training is routine [20, 23], yet a run can appear healthy through long optimization before abruptly collapsing, revealing little about what accumulated.”

“Training was conducted on a Google Colab Pro+ instance provisioned with an NVIDIA A100 (40GB), using BF16 mixed precision for faster training with stable numerics [27, 28].”

“Pure-bfloat16 training is attractive for memory but loses small updates to rounding [Kalamkar et al., 2019].”

“Next, we present the end-to-end inference latency of Lynx’s prototypes, comparing the results with uncompressed BF16 and the simple quantization schemes of INT8 and INT4.”

“Furthermore, we leverage BF16 mixed precision [28] and TF32 acceleration [8] under a global batch size of 64 (32 sequences per GPU) to balance numerical fidelity with computational through-put.”

“Throughout training, we adopt DeepSpeed ZeRO-3 [33] and FlashAttention V2 [11] with mixed-precision BF16 training [21].”

“This could easily be lessened with little to no impact on classification performance with bfloat16 quantization [24], while potentially improving classification speed on newer hardware optimized for bfloat16 operations.”

“To optimize training efficiency, we employed bfloat16 mixed precision [39], gradient checkpointing [40], and Flash Attention [41] acceleration techniques.”

“To further improve computational efficiency, BFloat16 precision [51] and FlashAttention-2 [52] were employed throughout the fine-tuning process.”

“However, while the reduced dynamic range of 16 bit can be compensated for by either switching to a BF16 representation with more exponent bits [15], or by introducing loss scaling , the further reduction to eight bits requires more sophisticated techniques.”

“To further reduce RAM usage, we adopt BF16 (brain floating point with 16 bits) [13] precision as an option to store hidden states throughout the SSM-subgraph.”

“To optimize computational efficiency, we utilized ZeRO Stage 3 [38] optimization for memory management and BFloat16 [45] mixed precision training.”

“While newer formats (e.g., bfloat16) are common in DL inference [27]–[29], they are not supported on the VCK190.”

“While several ligands such as FAD and ZN achieved high F 1 scores, others, such as BMA and MAN , achieved zero or near-zero predictive accuracy.”

“BF16 is also used in mixed-precision fine-tuning, as it shares the same exponent width as FP32, effectively eliminating gradient overflow [9].”

“We use standard acceleration techniques like bfloat16 (Kalamkar et al. 2019) and gradient checkpointing (Herrmann et al. 2019), diffusion training is conducted using full precision.”

“The industry widely quantizes both the model weights and the input values to low-bidwidth numbers such as BFLOAT16 [3], 8-bit and even 4-bit integers on CPUs and GPUs to reduce the memory usage, computational complexity, and power consumption [4]–[9].”

“To address the memory capacity challenges in training LLMs, there has been significant interest in compressing model parameters [11], [12], [33] and gradients [2], particularly for distributed training.”

“Without any acceleration techniques [6, 18, 21], both the design reviewer and contributor complete slide generation in under 30 seconds and require 8GB of VRAM using well under typical memory limits.”

“The Bfloat format is a low-precision numeric format proposed by Google Brain for deep learning training [23].”

“For optimization, the training procedure adopts the Adam (Kingma, 2014) optimizer ( β 1 = 0.9, β 2 = 0.999, ϵ = 1e-8) and BF16 (bfloat16) (Kalamkar et al., 2019) precision.”

“Specialized data types such as Google’s bFloat16 [20, 54], Intel’s Flexpoint [58] and NVIDIA’s TF32 [75] are also very effective.”

“Different floating-point formats, including FP16 (Micikevicius et al. 2018), BF16 (Kalamkar et al. 2019), FP8 (Micikevicius et al. 2022), HFP8 (Sun et al. 2019), and FP6 (Chmiel et al. 2020), along with variants such as HBFP (Drumond et al. 2018), S2FP8 (Cambier et al. 2020), and BM (Fox et al. 2021), have been propos…”

“However, as shown in [3], BF16-based CNNs offer the same accuracy as their FP32-based counterparts.”

“The resultant precision studies can see less than 5% accuracy loss for some models [5].”

“36–38 The most similar work to ours 39 uses RL and SAC to train different robotic tasks in simulated robotic environments, showing that neither FP16 nor mixed precision FP16 training with gradient scaling works effectively.”

“The Brain Floating Point (Bfloat16) [1] is a number system in which a 23-bit mantissa of FP32 is truncated to 7-bit while keeping the exponent and the sign bits unchanged.”

“5(b) further shows that LV can significantly speed up quantized FPs, too, e.g., from 1.35 to 2.21× (BF16).”

“For example, Google and Intel will flush the subnormal numbers of BF16 to 0.”

“Existing techniques utilize approximate computing uniformly, e.g., reducing the number of bits of an entire model or an entire layer [1].”

“…a 512-bit extension to the original 256-bit AVX and proposes vector instructions that support computations using 16-bit floats based on the IEEE half-precision format and the BFloat16 format [9], [10] that was proposed by Google for applications such as Convolutional Neural Network (CNN) inference.”