PipeDream: Fast and Efficient Pipeline Parallel DNN Training

“Due to space constraints we do not present end-to-end training results for AlexNet [26] and ResNet-50 [16] models.”

“Guided by prior work, we adjust the learning rate during training to converge to the desired result faster [31, 37, 38, 13, 24].”

“The first is the dataset for the Large Scale Visual Recognition Challenge 2012 (ILSVRC12) [32], also called the ImageNet 1K dataset.”

“, up to 85% of training time for VGG-16 [36]).”

“The first is the dataset for the Large Scale Visual Recognition Challenge 2012 (ILSVRC12) [32], also called the ImageNet 1K dataset.”

“When training Inception-v3 [19], VGG16 [36], Resnet-50 [16], and AlexNet [26] on the ILSVRC12 [32] dataset, PipeDream speeds up training by up to 1.”

“In our current implementation, we use Caffe [20] as the ML worker.”

“…five years has seen a rapid increase in the use of Deep Neural Networks (DNNs), with researchers and practitioners applying these models to great effect across a wide range of applications, including image and video classification, speech recognition, and language translation [16, 17, 21, 22, 44].”

“Guided by prior work, we adjust the learning rate during training to converge to the desired result faster [31, 37, 38, 13, 24].”

“The last five years has seen a rapid increase in the use of Deep Neural Networks (DNNs), with researchers and practitioners applying these models to great effect across a wide range of applications, including image and video classification, speech recognition, and language translation [16, 17, 21, 22, 44].”

“…five years has seen a rapid increase in the use of Deep Neural Networks (DNNs), with researchers and practitioners applying these models to great effect across a wide range of applications, including image and video classification, speech recognition, and language translation [16, 17, 21, 22, 44].”

“On one end, synchronizing at the end of every minibatch (referred to as bulk synchronous parallel or BSP [42]) reduces the staleness of weights used to compute gradients, ensuring good statistical efficiency.”

“…five years has seen a rapid increase in the use of Deep Neural Networks (DNNs), with researchers and practitioners applying these models to great effect across a wide range of applications, including image and video classification, speech recognition, and language translation [16, 17, 21, 22, 44].”

“For all the experiments, we measure the time taken to train the models until they reach their advertised validation accuracy : top-1 accuracy of 68% for VGG16, top-1 accuracy of 67% for Inception-v3, and METEOR [11] score of 0.294 for S2VT.”

“When training the S2VT [43] model on the MSVD [3] dataset, PipeDream speeds up training by 3x compared to data-parallel BSP.”

“Second, the burden of partitioning a model across multiple GPUs is left to the programmer [25], resulting in point solutions.”

“CNTK’s [33] 1-bit quantization technique tackles the problem of communication bottlenecks in data parallelism training [35].”

“Guided by prior work, we adjust the learning rate during training to converge to the desired result faster [31, 37, 38, 13, 24].”

“Goyal et al. [14] uses more efficient implementations of all_reduce , like the recursive halving-and-doubling algorithm and the bucket algorithm to reduce the amount of data being sent over the network [39].”

“Our experimental results corroborate recent findings that show that ASP does not reduce end-to-end DNN training time [9, 1, 4].”

“For machine learning (ML) problems such as matrix factorization, topic modeling, and linear regression, prior work [29, 27, 45, 23] has shown that model parallelism can often achieve faster training times than data parallelism because of the improved statistical efficiency that arises from not using…”

“In addition, determining how best to partition a DNN model among workers is a challenging task even for the most experienced machine learning practitioners [30], often leading to additional inefficiency.”

“Despite optimizations such as Wait-free Backpropagation [47], where weight gradients are sent as soon as they are available, (common in modern parameter servers), communication stalls are inevitable in data-parallel training due to the structure of the DNN computation, and the fact that communication can often dominat…”

“CNTK’s [33] 1-bit quantization technique tackles the problem of communication bottlenecks in data parallelism training [35].”

“Guided by prior work, we adjust the learning rate during training to converge to the desired result faster [31, 37, 38, 13, 24].”

“Each GPU maintains a full copy of the model and trains on its own partition of data while periodically synchronizing weights with other GPUs, using either collective communication primitives [14] or Parameter Servers [28, 9].”

“For machine learning (ML) problems such as matrix factorization, topic modeling, and linear regression, prior work [29, 27, 45, 23] has shown that model parallelism can often achieve faster training times than data parallelism because of the improved statistical efficiency that arises from not using…”

“Guided by prior work, we adjust the learning rate during training to converge to the desired result faster [31, 37, 38, 13, 24].”

“For machine learning (ML) problems such as matrix factorization, topic modeling, and linear regression, prior work [29, 27, 45, 23] has shown that model parallelism can often achieve faster training times than data parallelism because of the improved statistical efficiency that arises from not using…”

“This approximation strategy lacks generality and is effective for limited scenarios; it does not hurt convergence for some speech models [34], but hurts statistical performance due to noisy gradients in many others [9, 1].”

“Chen et al. [6] briefly explore the potential benefits of pipelining minibatches in model parallel training, but do not address the conditions for good statistical efficiency, scale, and generality as applicable to large real-world models.”

“The total training time to reach a desired accuracy level is simply the product of these two metrics [15].”

“[61] design PipeDream, a system that supports pipelined training and automatically determines how to systematically split a given model across the available computing nodes.”

“3D parallelism is actually a combination of three commonly used parallel training techniques, namely data parallelism, pipeline parallelism [178, 179], and tensor parallelism [65]10.”

“GPipe (concurrent work with an earlier PipeDream preprint [25]) uses pipelining in the context of model-parallel training for very large models [28].”

“Recently, PipeDream [22] introduced a pipelining approach to reduce communication overhead for synchronized training using parameter servers [34].”

“When a model does not fit in the device memory, model parallelism (MP) [7], [4], [11] and pipeline parallelism (PP) [12], [13] split the model among processes, in vertical and horizontal way respectively.”

“When a model does not fit in the device memory, model parallelism (MP) [7], [4], [11] and pipeline parallelism (PP) [12], [13] split the model among processes, in vertical and horizontal way respectively.”

“Such approaches are model parallelism (Krizhevsky et al., 2009), pipeline parallelism (Krizhevsky et al., 2009; Huang et al., 2018; Harlap et al., 2018), and operator parallelism (Lepikhin et al., 2020).”

“For instance, PipeDream is model parallelism through pipelined training presented in [368].”

“For graph-level partitioning, sophisticated algorithms [15, 24] are needed to reduce the overhead”

“Model Parallelism [50] partitions DNN models among GPUs to mitigate communication overhead and memory bottlenecks for distributed training [10], [14], [39], [40], [51]–[54].”

“For example, GPipe (Huang et al., 2019) and PipeDream (Harlap et al., 2018; Narayanan et al., 2019) push multiple inputs in sequence through a series of workers that each manage one model partition, allowing different workers to process different inputs in parallel.”

“Pipeline parallelism [17], [18], [19] splits a model vertically into pipeline stages and use micro-batching to hide pipeline bubbles.”

“To address this issue, the concept of pipeline parallelism [78], [79] has been introduced.”

“Current approaches that support pipeline parallelism are GPipe [70] and PipeDream [57, 58].”

“Pipeline parallelism [4, 7] involves partitioning model parameters and their activations across multiple devices through the division of models into pipeline stages.”

“Frameworks that support combinations of these strategies are generally designed for specific DNN applications [16, 22, 41, 50].”

“…include data parallelism (model weights replicated, but data sharded), tensor model-parallelism (Shazeer et al., 2018) (data and weight tensors are split across devices), and pipeline parallelism (Harlap et al., 2018; Huang et al., 2019) (entire layers or groups of layers are split across devices).”

“Parallelization strategies for distributed training are an active area of research [6], [45], [46], [54]– [56], and sometimes several strategies are jointly applied [18].”

“Harlap et al. (2018) propose using an asynchronous training algorithm to mitigate the effect of pipeline bubbles in microbach-based pipeline parallel training, but asynchronous training introduces uncertainty in model accuracy and is thus not widely adopted for training DNNs. Wavefront parallelism…”

“Two prominent examples of scale out training is model parallelism [5, 28] and pipeline parallelism [7,10], both partitioning the model states and the residual states across multiple GPUs.”

“PipeDream (Harlap et al., 2018) is a pipeline parallel distributed training technique used to reduce high communication-to-computation ratios.”

“According to the synchronization strategy we discussed earlier, gradients are aggregated by either synchronous pipeline model (GPipe [132]) or asynchronous pipeline model (PipeDream [115], SpecTrain [46], XPipe [107]).”

“A few prior works propose automatic methods for model partitioning [10] [23].”

“…from the challenge of classical distributed DNN training where (1) data samples are small yet the model is large (model parallelism (Krizhevsky, 2014; Harlap et al., 2018)) and (2) data samples do not have dependency (data parallelism (Li et al., 2020; 2018b;a)), both violating the nature of GCNs.”

“Furthermore, [13] may suffer from staleness of the weight update, which is linear in the number of stages.”

“Often, a combination of both approaches is employed [33, 16].”

“Another direction is to partition a large model along its layers across multiple GPUs and then stream in small data batches through the layer pipeline, e.g., PipeDream (Harlap et al., 2018) and PipeMare (Yang et al., 2021).”

“Techniques like tensor-slicing (Shoeybi et al., 2020) and pipeline parallelism (Harlap et al., 2018) designed to scale dense models are not effective in scaling the number of experts due to the sparse nature of MoE models.”

“3D parallelism combines data parallelism [2, 6], pipeline parallelism [13, 14, 21] and tensor parallelism [32] to distribute model training workloads across hundreds of GPUs.”

“With the aim of further improving the communication efficiency of FL in terms of accelerating the entire training process, some works on FL with sparsity have considered making use of the models’ layered structure to parallelize the communications with computations, which is referred to the pipeline [71, 72, 73].”

“The 4D hybrid parallelism includes data parallelism (DP) [60, 61], intra-layer tensor model parallelism (MP) [62, 57], inter-layer pipeline model parallelism (PP) [32, 63, 64, 65, 33], and the improved sharded data parallelism (Sharded) based on ZeRO [58].”

“At every iteration, workers must synchronize their model parameters by exchanging and aggregating their gradients to ensure convergence [39].”

“Though the model parallelism has been a popular way in implementing DNN training on multiple devices in a distributed manner [12, 15, 22, 9, 39, 45, 56], it is important to note that the devices on which the training is distributed usually refer to computing units like GPUs connected to one server cluster through high…”

“oss multiple GPUs, and each GPU computes only a part of the model. A problem of model parallel training is underutilization of GPUs due to the small size of each fragment assigned to a GPU. PipeDream [15] addresses the problem using overlapped computation for 13 multiple iterations. However, the staleness caused by co…”

“There are a number of schemes to achieve parallelism among multiple workers in distributed training: data parallelism [44, 50], model parallelism [9], hybrid parallelism [28, 48, 49, 57, 58], and pipeline parallelism [17].”

“Among the earliest efforts were GPipe [12] and PipeDream [10], which remains one of the most widely used Pipeline methods.”

“…AccPar [222], Whale [223], AMP [224], Pathways [225], HPH [226], SDPIPE [227], HAP [228], PipePar [229], Yuan et al. [230], SWARM [231], FusionAI [232] Heterogeneous Model DeepSpeed-Chat [233], HuggingFace TRL [234], OpenRLHF [235], Adpative Placement and Parallelism [236], ReaLHF [237],…”

“[2019] in combination with pipeline parallelism [Harlap et al., 2018] to distribute the model across GPUs.”

“To handle significant sequence length variation across text, image, and video data, we employ a heuristic scheduling strategy to reduce imbalance bubble across pipeline-parallel (PP) [6, 9–11, 19, 20] and data-parallel (DP) groups.”

“Recent efforts have also looked into hybrid [49] and pipelined [36, 38] parallelization strategies.”

“One line of work that improves over GPipe focuses on asynchronous PP, including PipeDream (Harlap et al., 2018), PipeMare (Yang et al., 2021).”

“It is benefitting to combine different types of parallelization strategies [16, 14, 26, 12, 9, 11, 7], and recent lines of research questions how to find an optimal strategy”

“Among them, interlayer partition [10, 11] often requires the model to be explicitly expressed as a stack of sub-modules, the number of which equals to the number of processors, and would introduce either gradient staleness or bubble overhead; while the sparse activation [12] is primarily designed for Google TPU cluste…”

“Pipelined parallelism [7, 9] is proposed to split the whole model by layer, which will be executed in a pipeline fashion.”

“PipeDream [12] proposes a DP algorithm that splits the computation graph into balanced subgraphs.”

“Very few papers combine pipelining, function parallelism, and data parallelism [74].”

“The parallelism of communication and computing mainly aims to optimize the order of computation and communication such that the communication cost can be minimized [63][64][65][66][67][68][69][70].”

“Forward and backward passes can be interleaved at each processing element to form a bidirectional training pipeline [30].”

“[15] proposed PipeDream, a training system that parallelizes computations by pipelining executions.”

“Optimizations are proposed for both operation partitioning approach [37, 74, 75, 92] and pipeline parallel approach [32, 34, 84].”

“Such an issue can be addressed by either increasing EPC size with specialized SGX card [13], or applying existing model parallelism techniques to split large models [25, 29].”

“In addition to this, despite adopting the most advanced learning policies and employing high-end GPU clusters, complex CapsNets and CNNs typically require long training time [5][19].”

“This work is motivated by the state-of-the-art synchronous pipeline approach GPipe [14] as well as asynchronous pipeline approaches PipeDream [15] and SpecTrain [16], which will be detailedly reviewed in the next section.”

“Itisassumedthatthepipelinesupportsasynchronouscommuni-cation[12],[13],allowingforexactoverlapofcomputationand communication.”

“…as prevailing large decoder-only AR LMs (Brown et al., 2020; Touvron et al., 2023a;b; Jiang et al., 2023) while incorporating extra encoder adds to the complexity in key techniques for scaling LLMs such as pipeline parallelism (Harlap et al., 2018; Huang et al., 2019; Li & Hoefler, 2021).”

“Harlap et al. [58] combine pipelining, data parallelism, and model parallelism, called pipeline parallelism, and develops a DNN training system, PipeDream.”

“PipeDream [18] introduces a pipelining approach to overlap communication and computation for asynchronous training.”

“If weight stashing (Harlap et al., 2018) is used to mitigate weight inconsistency the resulting update is: Weight stashing requires the overhead of storing parameter versions along with the activations.”

“and another one in the parallelization of the DL model [12], [13].”

“We choose PipeDream [27] as our baseline to compare the performancewiththeproposeddynamicschedulingmechanism.”

“Pipelining (Harlap et al., 2018) concurrently passes multiple mini-batches to multiple layers on multiple devices.”

“…a combination of four types of parallelisms, including Data Parallelism (DP) (Li et al., 2020), Tensor Parallelism (TP) (Shoeybi et al., 2019), Sequence Parallelism (SP) (Korthikanti et al., 2023), and Pipeline Parallelism (PP) (Huang et al., 2019a; Harlap et al., 2018; Narayanan et al., 2021).”

“Note that pipeline is not a new idea [33], [34], however, we draw an analogy between the stages in GPipe and different phases of MoE dataflow, then we introduce pipeline parallelism for MoE. for a micro-batch, experts initiate asynchronous computations while concurrently beginning to receive another…”

“Model Parallelism splits the model parameters over multiple processors either to improve parallelization [6] or when it is too large to fit in the memory of a single device [30, 44, 73].”

“Pipeline parallelism (Huang et al., 2019; Harlap et al., 2018; Narayanan et al., 2021) Offloading.”

“One is a classic ResNet [44] distributed training in PyTorch with the widely-used 1F1B mechanism [43, 101] that tries to overlap communication with computation.”

“Model parallelism includes two categories: pipeline parallelism [18], [20], [21], which assigns individual layers to single GPUs, and tensor parallelism [22], [23], [24], which divides each tensor into chunks allocated to specific GPUs.”

“In addition, recent research [16, 39, 51] has done great deal of work on performance improvement on multi-GPU environments.”

“When the model cannot fit into the device memory, model parallelism [23], [24] and pipeline parallelism [25], [7] can be leveraged to split the model layers or parameters to make the best use of the memory across multiple devices.”

“Although some works on asynchronous pipeline parallelism proposed various techniques, including weight stashing [8], to mitigate the negative effects of parameter staleness and showed conver-gences for relatively small models, other works pointed out that high numerical stability was required for…”

“Harlap et al. [85] suggested hybrid models in parallelism.”

“TP and PP belong to model parallelism, where PP [24,26] partitions different model layers on different GPUs.”

“Recently, independent work on PipeDream [5] proposed a distributed pipeline system for DNN training.”

“separate device, and micro-batching is employed to conceal any pipeline bubble [278].”

“Under this background, model parallelism [Harlap et al., 2018, Huang et al., 2019, Shoeybi et al., 2019, Zheng et al., 2022] is proposed to partition parameters into a set of devices to address the memory constraint.”

“More recently studied approaches combine data parallelism and model parallelism in a pipelined fashion [8].”

“GPipe and PipeDream both have overheads related during the start of the pipeline, and both approaches require the number of devices to scale with the model depth and not just the layer size.”

“…setups, such as the example illustrated in Figure 1, rely on domain-specific stacks [1, 37, 49, 50, 63], scalable frameworks such as Megatron [64], DeepSpeed [59], and NeMo [30], which introduce techniques for parallelism, memory management, and throughput optimization [12, 18, 57, 59, 62, 70].”

“, 2020), pipeline-parallel training (Huang et al., 2019; Harlap et al., 2018) and model sharding (Ren et al.”

“To address the issue of high memory demand, one-forward-one-backward (1F1B) scheduling strategies are proposed [14, 15, 9] to rewrite GPipe to schedule backward passes in advance and make backward passes have higher execution priority than forward passes, without affecting final results.”

“Works like PipeDream [27], GPipe [31], DAPPLE [22], PipeMare [79] introduce and optimize various aspects of Pipeline Parallelism.”

“Although there is huge amount of work on model-distributed training [1], [2], [3], [4], the benefit of model distribution for inference is not understood well.”

“Beyond model-parallel and data-parallel approaches, certain frameworks like PipeDream have also explored whether breaking apart an ML pipeline, technically a DNN, may be more efficient, where individual machines are dedicated to the training of individual ML pipeline segments [150].”

“Pipeline Parallelism (PP) [Huang et al. , 2019; Harlap et al. , 2018], different from TP, treats the model as a sequence of layers and partitions them into multiple stages across devices to minimize the memory cost, while consistent communication of the intermediate results is necessary to complete…”

“Specifically, ZeRO++ integrates blocked-based quantization kernels [29] into model weights and gradient communications to drastically reduce message size.”