PipeDream: Fast and Efficient Pipeline Parallel DNN Training

“, up to 85% of training time for VGG-16 [36]).”

“Due to space constraints we do not present end-to-end training results for AlexNet [26] and ResNet-50 [16] models.”

“Guided by prior work, we adjust the learning rate during training to converge to the desired result faster [31, 37, 38, 13, 24].”

“The first is the dataset for the Large Scale Visual Recognition Challenge 2012 (ILSVRC12) [32], also called the ImageNet 1K dataset.”

“The first is the dataset for the Large Scale Visual Recognition Challenge 2012 (ILSVRC12) [32], also called the ImageNet 1K dataset.”

“When training Inception-v3 [19], VGG16 [36], Resnet-50 [16], and AlexNet [26] on the ILSVRC12 [32] dataset, PipeDream speeds up training by up to 1.”

“In our current implementation, we use Caffe [20] as the ML worker.”

“…five years has seen a rapid increase in the use of Deep Neural Networks (DNNs), with researchers and practitioners applying these models to great effect across a wide range of applications, including image and video classification, speech recognition, and language translation [16, 17, 21, 22, 44].”

“Guided by prior work, we adjust the learning rate during training to converge to the desired result faster [31, 37, 38, 13, 24].”

“The last five years has seen a rapid increase in the use of Deep Neural Networks (DNNs), with researchers and practitioners applying these models to great effect across a wide range of applications, including image and video classification, speech recognition, and language translation [16, 17, 21, 22, 44].”

“…five years has seen a rapid increase in the use of Deep Neural Networks (DNNs), with researchers and practitioners applying these models to great effect across a wide range of applications, including image and video classification, speech recognition, and language translation [16, 17, 21, 22, 44].”

“On one end, synchronizing at the end of every minibatch (referred to as bulk synchronous parallel or BSP [42]) reduces the staleness of weights used to compute gradients, ensuring good statistical efficiency.”

“…five years has seen a rapid increase in the use of Deep Neural Networks (DNNs), with researchers and practitioners applying these models to great effect across a wide range of applications, including image and video classification, speech recognition, and language translation [16, 17, 21, 22, 44].”

“For all the experiments, we measure the time taken to train the models until they reach their advertised validation accuracy : top-1 accuracy of 68% for VGG16, top-1 accuracy of 67% for Inception-v3, and METEOR [11] score of 0.294 for S2VT.”

“When training the S2VT [43] model on the MSVD [3] dataset, PipeDream speeds up training by 3x compared to data-parallel BSP.”

“Second, the burden of partitioning a model across multiple GPUs is left to the programmer [25], resulting in point solutions.”

“Goyal et al. [14] uses more efficient implementations of all_reduce , like the recursive halving-and-doubling algorithm and the bucket algorithm to reduce the amount of data being sent over the network [39].”

“Guided by prior work, we adjust the learning rate during training to converge to the desired result faster [31, 37, 38, 13, 24].”

“Our experimental results corroborate recent findings that show that ASP does not reduce end-to-end DNN training time [9, 1, 4].”

“For machine learning (ML) problems such as matrix factorization, topic modeling, and linear regression, prior work [29, 27, 45, 23] has shown that model parallelism can often achieve faster training times than data parallelism because of the improved statistical efficiency that arises from not using…”

“CNTK’s [33] 1-bit quantization technique tackles the problem of communication bottlenecks in data parallelism training [35].”

“CNTK’s [33] 1-bit quantization technique tackles the problem of communication bottlenecks in data parallelism training [35].”

“In addition, determining how best to partition a DNN model among workers is a challenging task even for the most experienced machine learning practitioners [30], often leading to additional inefficiency.”

“Guided by prior work, we adjust the learning rate during training to converge to the desired result faster [31, 37, 38, 13, 24].”

“Each GPU maintains a full copy of the model and trains on its own partition of data while periodically synchronizing weights with other GPUs, using either collective communication primitives [14] or Parameter Servers [28, 9].”

“Despite optimizations such as Wait-free Backpropagation [47], where weight gradients are sent as soon as they are available, (common in modern parameter servers), communication stalls are inevitable in data-parallel training due to the structure of the DNN computation, and the fact that communication can often dominat…”

“For machine learning (ML) problems such as matrix factorization, topic modeling, and linear regression, prior work [29, 27, 45, 23] has shown that model parallelism can often achieve faster training times than data parallelism because of the improved statistical efficiency that arises from not using…”

“Guided by prior work, we adjust the learning rate during training to converge to the desired result faster [31, 37, 38, 13, 24].”

“Chen et al. [6] briefly explore the potential benefits of pipelining minibatches in model parallel training, but do not address the conditions for good statistical efficiency, scale, and generality as applicable to large real-world models.”

“This approximation strategy lacks generality and is effective for limited scenarios; it does not hurt convergence for some speech models [34], but hurts statistical performance due to noisy gradients in many others [9, 1].”

“For machine learning (ML) problems such as matrix factorization, topic modeling, and linear regression, prior work [29, 27, 45, 23] has shown that model parallelism can often achieve faster training times than data parallelism because of the improved statistical efficiency that arises from not using…”

“The total training time to reach a desired accuracy level is simply the product of these two metrics [15].”

“[61] design PipeDream, a system that supports pipelined training and automatically determines how to systematically split a given model across the available computing nodes.”

“3D parallelism is actually a combination of three commonly used parallel training techniques, namely data parallelism, pipeline parallelism [178, 179], and tensor parallelism [65]10.”

“GPipe (concurrent work with an earlier PipeDream preprint [25]) uses pipelining in the context of model-parallel training for very large models [28].”

“Recently, PipeDream [22] introduced a pipelining approach to reduce communication overhead for synchronized training using parameter servers [34].”