PipeDream: Fast and Efficient Pipeline Parallel DNN Training

“, up to 85% of training time for VGG-16 [36]).”

“Due to space constraints we do not present end-to-end training results for AlexNet [26] and ResNet-50 [16] models.”

“Guided by prior work, we adjust the learning rate during training to converge to the desired result faster [31, 37, 38, 13, 24].”

“The first is the dataset for the Large Scale Visual Recognition Challenge 2012 (ILSVRC12) [32], also called the ImageNet 1K dataset.”

“The first is the dataset for the Large Scale Visual Recognition Challenge 2012 (ILSVRC12) [32], also called the ImageNet 1K dataset.”

“When training Inception-v3 [19], VGG16 [36], Resnet-50 [16], and AlexNet [26] on the ILSVRC12 [32] dataset, PipeDream speeds up training by up to 1.”

“In our current implementation, we use Caffe [20] as the ML worker.”

“…five years has seen a rapid increase in the use of Deep Neural Networks (DNNs), with researchers and practitioners applying these models to great effect across a wide range of applications, including image and video classification, speech recognition, and language translation [16, 17, 21, 22, 44].”

“Guided by prior work, we adjust the learning rate during training to converge to the desired result faster [31, 37, 38, 13, 24].”

“Specifically, ZeRO++ integrates blocked-based quantization kernels [29] into model weights and gradient communications to drastically reduce message size.”

“On the other hand, Pipeline Parallelism (PP) [10], [13], [14] adopts a horizontal model splitting approach, assigning different layers to different devices.”

“PipeDream [22] introduces a pipeline approach to reduce communication overhead for synchronized training with the parameter server architecture [31].”

“…parallelism (Shoeybi et al., 2019; Narayanan et al., 2021; Korthikanti et al., 2023), ZeRO data parallelism (Rajbhandari et al., 2020), PipeDream pipeline parallelism (Harlap et al., 2018), and more specifically, expert parallelism (Lepikhin et al., 2021) by combining data and tensor parallelism.”

“With the increase in cluster size driven by larger models and data, communication has often been such a bottleneck, and many new training algorithms have thus emerged (Shazeer et al., 2017; Fedus et al., 2022).”

“This scheme is similar to Huang et al. (2019), but differs from Harlap et al. (2018) in the detail that we recompute each forward stage before its respective backwards pass.”

“To address this problem, parallelism strategies such as pipeline parallelism (PP) [30, 31] and data parallelism (DP) [11, 16] have been developed to distribute models across multiple GPUs and increase throughput for inference and training.”

“PipeDream [20] brings the pipeline framework into deep learning passes in practice.”

“GPipe [6] and PipeDream [5] split the model vertically.”

“2018), pipeline parallelism (Harlap et al. 2018; Huang et al. 2019) and data parallelism (Tarditi, Puri, and Oglesby 2006).”

“Our strategy leverages a combination of 16-way Pipeline Parallelism (PP) with virtual stages [29, 54, 39, 58, 48, 22], 16-way Expert Parallelism (EP) [40], and ZeRO-1 Data Parallelism [61].”

“Between them, self-attention is the core of the transformer, enabling LLMs to capture long-range contextual dependencies.”

“The results show that MPipeMoE achieves 2.8 × speedup and reduces memory footprint by up to 47% compared to FasterMoE.”

“PipeDream [8] and DAPPLE [7] further reduce the activation memory consumption by proposing the 1F1B ( one-forward-one-backward ) scheduling method, as shown in Figure 2.”

“Pipeline parallelism [46, 47, 48] involves partitioning the model depth-wise across GPUs, which enables efficient pipeline scheduling.”

“PipeDream [10] combines traditional data parallelism with model parallelism enhanced with pipelining.”

“Despite their successes, these large-scale models necessitate extensive GPUs for parallel training, employing strategies like data parallelism [11], pipeline parallelism [12], tensor parallelism [13].”

“PipeDream partitions layers of model to place them on all available GPUs while achieving its dual goal of balancing computation and minimizing communication [12].”

“PipeDream [99, 195] further refines the PP approach with its One Forward pass followed by One Backward pass (1F1B) strategy.”

“To address this limitation, several strategies, including various models of parallelism that utilize the resources of multiple GPUs, have been proposed [34–37].”

“Furthermore, DL parallelism approaches tend to have consistent communication overheads per minibatch [2, 41].”

“[11] aims to €nding the best solutions of PP in a synchronous way and [12, 13] tries to search the OPP.”

“Similarly, communication delay problems arise in workloads from MapReduce systems and in the model parallelism paradigm [16] for training complex machine learning models on large clusters; see [16, 41, 20, 42] and references therein.”

“An early notable work is 1F1B (Fan et al., 2021; Harlap et al., 2018), which uses a one-forward-one-backward pattern to mitigate the high memory usage of GPipe (Huang et al., 2019).”

“A pipelined architecture can be considered as in [4] to improve training throughput.”

“A promising approach might be to parallelize simulation across neuron populations, similarly to how PipeDream [42] parallelizes backpropagation across layers.”

“Pipeline parallelism (PP) aims to solve the memory constraints of DP by leveraging the layer-wise nature of DNNs[10, 11].”

“Finally, a pipelined architecture can be considered, to improve the training throughput [21].”

“Most DL architectures are dominated by high-intensity matrix operators, and standard tooling for DL training optimization has evolved to support models that fit this pattern [9, 12, 14, 19, 29, 42, 44].”

“We model this as min ||||| v∑ i=u (1 − xi) × tlcomp − Cost) ||||| , (16) s.t. ( v∑ i=u xi × mai + m p i ) + max u≤i≤j≤v,xi=1,xj=1,∀k⊆[i,j] ,xk=0 j−1∑ k=i+1 xk × mak ≤ d × MEMG, (17) xi𝜀{0,1}.”

“Additionally, the additional input layer in the first stage exacerbates the memory bottleneck for most PP schedules like 1F1B (Harlap et al., 2018).”

“It allows distributed systems to use resources efficiently by splitting each batch into micro-batches, which are then pipelined on the block-divided model as in, for example, Gpipe [12], Xpipe [13], PipeDream [14].”

“On the other hand, pipeline-parallel frameworks like PipeDream [45] do not eliminate communication, but interleave computation with communication by breaking training mini-batches into micro-batches.”

“For example, the 1F1B pattern proposed by PipeDream [21] runs one forward stage for every backward stage (on different microbatches) tomaintain a perfect ratio and utilization.”

“The real-time response is also guaranteed by leveraging the computing re-sourcesofmultipleedgeservers.”

“A re-built version of Figure 8 from PipeDream Harlap et al. (2018).”

“GPipe [11] and Pipedream [12] propose pipeline parallelism to further improve model parallelism, by pipelining forward computation and backward propagation across several microbatches.”

“GPipe [8] and DAPPLE [1] are representative works of synchronous pipeline-based parallelism of DL while asynchronous approaches include PipeDream [5].”

“PipeDream has reduced communication up to 95% and is up to 5x faster compared with data parallel training [16].”

“We proposed combined traditional data parallelism with model parallelism enhanced with pipelining [39].”

“We used 32 because it is common to use for inspecting the behavior of DNNs [23], but the trends in the time estimates remained the same with other batch sizes.”

“To allow concurrent execution, we release the Python GIL lock within the CPU adam thread, allowing the primary Python thread to continue processing.”

“Our work builds upon and extends prior research in parallelization strategies, long-sequence processing, and communication optimization.”

“However, the synchronous design of GPUs cannot fully exploit the event-driven and sparse-asynchronous advantages of spiking signals [11, 12].”

“Extending the framework to support more advanced LLM architectures and serving techniques, such as mixture of experts (MoE) [17] and pipeline parallelism (PP) [14, 16], would greatly expand its applicability.”

“…using the loss of both of the buffers does not have a particular effect on performance compared to the delay, further justifying our approach. approaches (Harlap et al., 2018; Xu et al., 2019; Kosson et al., 2021), we emphasize the practical memory savings associated with different methods in Tab.”

“This illustration is proposed in PipeDream [34]. stage.”

“We train all models on Graphcore Mk2 IPU clusters using a combination of pipeline model parallelism and data parallelism (Harlap et al., 2018).”

“Model parallelism examples are PipeDream [28], Megatron-LM [29], and GPipe [30].”

“argue that the staleness produced by their method does not affect the update step for a vanilla SGD optimizer [11], such argument would be invalid when combined with other techniques commonly used in first-order optimizers (such as momentum in Adam [17]).”

“Similar staleness effects appear in distributed optimization and parallel training [34, 35].”

“…plain computational units that can be split into parallel computing units with balanced loads across distribution training nodes, the discriminative series RecSys model computations workflow is too complex, making it difficult to leverage state-of-the-art parallel computing techniques [14, 17].”

“Pipelining is a promising solution to address this problem by keeping workers busy [25], [28], [29].”

“Graph partitioning [15, 16, 17, 19, 23, 6, 48] splits the training data into micro-batches and they are pipelined into the training devices.”

“Other works such as PipeDream explore asynchronous pipelining [15, 32, 49, 11, 27, 43, 34], a memory-efficient alternative to standard pipelining.”

“In fact, synchronous methods achieve better results regarding the accuracy of the CNN models compared to the asynchronous approaches [13], [32], particularly, with a short synchronization period [33].”

“While asynchronous PP schemes [91], [92] can eliminate pipeline bubbles, they have no guarantee of model convergence [93].”

“Parallel strategies are used to train large models at scale [15, 19, 25, 46–48, 54].”

“PipeDream [7], [8] eliminates idle workers and thus improves efficiency via a dedicated scheduling and batching mechanism.”

“Pipeline parallelism [7, 8, 41] segments a model into several stages and distributes them across different devices.”

“Recent studies on pipelined model parallelism, such as those presented in [20, 22, 23], highlight the need for model parallelism in handling large DNNs that may not fit on a single machine’s memory.”

“The most popular techniques include data parallism [11, 12], model parallism [13, 14], pipeline parallism [15, 16] and hybrid parallism [14, 17], which aim to accelerate the training process from different dimensions.”

“PipeDream [18] combined traditional data parallelism with model parallelism enhanced with pipelining.”

“WFBP, as one of the earliest structure-aware scheduling for distributed DL, has inspired a few follow-up work, such as ByteScheduler [52] and PipeDream [30].”

“Interleaving microbatch computations, and memorizing the result of calculations were proposed to parallelize this process [10], [11].”

“The increased delay tolerance that AB provides could enable hardware speedups for both data-parallel distributed training as well as pipeline-parallel training with pipelined backpropagation (Pétrowski et al., 1993; Chen et al., 2012; Harlap et al., 2018).”

“) search, which itself is a compute-intensive task, 2) DataParallel training has a synchronization (allreduce) overhead that increases linearly with respect to the number of processes [16, 19, 22], and 3) All DNN related data has to fit inside the device (CPU/GPU) memory.”

“This issue is recently recognized and investigated in areas such as distributed deep learning or parallel deep neural networks [44].”

“Distributed attention exploits sequence parallelism , a new parallelism that is additional to data parallelism [12], tensor parallelism [22], and pipeline parallelism [13,14] for LLM training.”

“Yet, pipelining is always done in a sequential execution of all layers (Huang et al., 2019; Qi et al., 2024; Harlap et al., 2018; Park et al., 2020).”

“Moreover, the role of GPU computing has become increasingly pivotal in the training of LLMs[5].”

“Enhanced versions include adjusted averaging mechanisms and approximation functions to handle staleness, enabling ASGD to rival synchronous methods in non-convex optimization problems [58, 17, 50].”

“Meanwhile, we focus more on edge computing path selection for reduced communication overhead but does not like [28, 29] for hardware architecture design.”

“There are several possible ways of scheduling forward and backward microbatches across devices, either in a synchronous way such as GPipe [47] or in an asynchronous way such as PipeDream [48].”

“Pipeline Parallelism: To accelerate the distributed training process, PipeDream (Harlap et al., 2018; Narayanan et al., 2019, 2021) and GPipe (Huang et al., 2019) propose the pipelined model parallelism so that multiple input data can be pushed through all the available workers in a sequential order.”

“They [11] introduce a paralleled computation by pipelining execution across multiple machines, which show 95% for large deep neural networks relative to data-parallel training.”

“THE centralized parameter server (PS) architecture [1], [2], [3] has achieved great success to operate large-scale distributed Deep Learning (DL) tasks [4], [5], [6], [7], [8], [9]”

“We support this assertion with three case study systems: FlexRR, Proteus, and PipeDream.”

“Several works have been proposed [15, 16, 20] to make model parallelism more efficient by introducing different kinds of pipeline mechanism.”

“When using PD disaggregation, each phase can use a different number of GPUs and a different parallelism strategy, such as tensor parallelism (TP) [51] or pipeline parallelism (PP) [24, 33].”

“Mini-batches are split into micro-batches and passed through the pipeline, allowing different devices to work on different parts of the input simultaneously, improving hardware utilization [11, 26, 27].”

“On the other hand, Pipeline Parallelism (PP) [37], [38] adopts a horizontal model splitting approach, assigning different layers to different devices.”

“Input mini-batches are split into micro-batches to increase utilization through pipelined execution [9, 17, 20, 29, 41].”