A Study of BFLOAT16 for Deep Learning Training

“When training ResNet-50 on Imagenet using the AVX512BF16 instructions, we achieved a Top-1 accuracy of 75.62%, which matches the current state-of-the-art performance.”

“The network uses a “VGG loss” function using a pre-trained VGG19 network [33].”

“Here we choose AlexNet (ILSVRC 2012) [22] and ResNet-50 (ILSVRC 2015) [18] as representative models for the BFLOAT16 evaluation.”

“However, all the aforementioned methods are shown to work for smaller benchmark model/data-sets only and invariably result in a non-trivial drop in accuracy with larger ImageNet data-set [9] and classification task [32].”

“However, all the aforementioned methods are shown to work for smaller benchmark model/data-sets only and invariably result in a non-trivial drop in accuracy with larger ImageNet data-set [9] and classification task [32].”

“Using multiple deep learning frameworks modified to insert appropriate Quantlib calls, we provide SOTA results for: AlexNet [22], ResNet-50 [18], DC-GAN [29], SR-GAN [24], DeepSpeech2 [4], GNMT [37], and two industrial workloads namely: a Deep and Cross Network, and a DNN Recommendation System.”

“We have implemented DC-GAN in Caffe and for our experiments all the input tensors (activations, weights) are converted to BFLOAT16 for convolution layers (in both the generator and the discriminator), while only the input activations are converted to BFLOAT16 for batch normalization layers; all other tensors are maint…”

“There are at least three half-precision formats in the domain of mixed precision training of large neural networks: FP16 [28], 16-bit Integer based [9], and BFLOAT16 [10, 4, 3].”

“The Julia language [5], which is designed to provide high-performance without sacrificing ease of programming, has also been shown to be benefited from BFLOAT16 [11].”

“A popular format showing little loss in overall precision is bfloat16, which is currently preferred to the more standard IEEE 16-bit half-precision floating-point format in machine learning applications [31].”

“For 16-bit encodings ( N “ 2 16 ), we use bfloat16, a standard which is commonly used in machine learning [4].”

“Suppose, for example, we use b = 4 bits for representing the weights and activations, and employ a B = 32 bit accumulator, as common in modern architectures (Kalamkar et al., 2019; Rodriguez et al., 2018).”

“With half the number of bits used compared to FP32, use of BF16 can theoretically double the performance without significant accuracy loss in deep learning applications [24, 25].”

“Google introduced a new custom floating-point format called "bfloat16" a less precise version of IEEE 754 single-precision floating-point number format [16].”

“One limitation of this study is that the empirical exploration focuses on microprocessor (FP32) and integrated-GPU (FP16) specific acceleration, and thus future work involves studying the impact of architecture design on hardware-specific acceleration for different hardware designs such as INT8 and BFloat16 [17] accel…”

“Además del IEEE 754 [8,9], han de notarse bfloat16 [10], el formato logaŕıtmico [11,12] y los posits [7,13,14].”

“New data representation formats targeting dynamic and low accuracy opportunities for deep learning have been proposed [13], [33].”

“Among those, bfloat16, which was conceived for DL training, has become the numeric alternative because of its wider dynamic range and smaller footprint [1, 10, 23, 25].”

“In classical deep learning, mixed precision arithmetic has emerged as a standard performance optimization technique, enabling substantial reductions in memory footprint and computational time while maintaining convergence behavior [12, 7].”

“Previous studies have shown that neural networks trained and evaluated using BFLOAT16 can achieve accuracy comparable to FP32 baselines across a wide range of architectures and tasks, confirming that reduced precision is often sufficient for practical AI workloads [11,19,2,10].”

“In each step, each bank fetches the corresponding matrix subchunk via column accesses and performs 16 parallel BFLOAT16 [22] multiplications with the current input-vector subchunk.”

“The NVIDIA A100 GPUs that we ultimately had access to support the bfloat16 format (Wang and Kanwar, 2019; Kalamkar et al., 2019), which has the same dynamic range as float32.”