PipeDream: Fast and Efficient Pipeline Parallel DNN Training

“, up to 85% of training time for VGG-16 [36]).”

“Due to space constraints we do not present end-to-end training results for AlexNet [26] and ResNet-50 [16] models.”

“Guided by prior work, we adjust the learning rate during training to converge to the desired result faster [31, 37, 38, 13, 24].”

“The first is the dataset for the Large Scale Visual Recognition Challenge 2012 (ILSVRC12) [32], also called the ImageNet 1K dataset.”

“The first is the dataset for the Large Scale Visual Recognition Challenge 2012 (ILSVRC12) [32], also called the ImageNet 1K dataset.”

“When training Inception-v3 [19], VGG16 [36], Resnet-50 [16], and AlexNet [26] on the ILSVRC12 [32] dataset, PipeDream speeds up training by up to 1.”

“In our current implementation, we use Caffe [20] as the ML worker.”

“…five years has seen a rapid increase in the use of Deep Neural Networks (DNNs), with researchers and practitioners applying these models to great effect across a wide range of applications, including image and video classification, speech recognition, and language translation [16, 17, 21, 22, 44].”

“Guided by prior work, we adjust the learning rate during training to converge to the desired result faster [31, 37, 38, 13, 24].”

“To tackle these challenges, we propose a shift in the application scenario from the cloud to the edge, along with the extension of the pipeline parallel method [8] from DNN training to DNN inference.”

“Typically, parallelism in LLM computation includes data parallelism (Rajb-handari et al., 2020), pipeline parallelism (Harlap et al., 2018; Huang et al., 2019), and tensor parallelism.”

“Although every edge device has limited resources, multiple edge devices Horizontal partition borrows the idea of naive model parallelism [24], [25] in the distributed training.”

“The majority of these approaches can be classified into three categories: (Cun et al. 1990), Optimal Brain Surgeon (Hassibi et al. 1993), hard-thresholding (Han et al. 2015), and similar works (Castellano, Fanelli, and Pelillo 1997; Leung et al. 2001) remove the insignificant edges or neurons by considering the magnit…”

“The work in [25] proposes PipeDream, a pipelined training framework that automatically finds the best way to split an input model across the available edge nodes.”

“Model parallelism presented in [15] is another method to accelerate the training time on CNNs.”

“Nonetheless, within the scope of distributed deep learning, conventional approaches [10], [11] encounter obstacles attributed to dynamic workloads [12], tightly coupled operations [13], and synchronized node patterns, thereby diminishing the efficacy of standard logging and tracing mechanisms.”

“Mini-batches are split into micro-batches and passed through the pipeline, allowing different devices to work on different parts of the input simultaneously, improving hardware utilization [11, 27, 28].”

“Pipeline Parallelism (PP) distributes the DNN across the dimen-sion of the ML layer [14, 15].”

“This either leads to weight version mismatches between forward and backward computations (such as PipeMare[29]) or introduces weight staleness if weight version consistency is respected (such as PipeDream[8, 20] and PipeDream-2BW[21]).”

“Recent efforts have also looked into hybrid [19] and pipelined [11], [13] parallelization strategies.”

“PP, investigated in systems such as GPipe [4] and PipeDream [5], addresses memory scalability by partitioning model layers across sequential pipeline stages.”

“Most existing PP approaches in LLM training focus on minimizing pipeline bubbles or reducing activation memory usage [23], [24], [25], [26], [27], [28], [29], [30], [31].”

“The complete forward pass F and backward pass B are then compositions across all layers [27,28]: Pipeline parallelism partitions the L layers into D contiguous, mutually exclusive groups called stages, one per GPU [27,28]: Each GPU updates the parameters of its stage.”

“Inter-Operator Decomposition (InterDecom) achieves overlap by partitioning both computation and communication operators into d smaller, independent sub-operators that can be pipelined [4], [13], [14], [20].”

“Motivated by this, the primary objective of this study is to thoroughly analyze, through an in-house simulator, the opportunities and challenges associated with data [67] and model parallelism techniques, such as tensor [72] and pipeline [29] parallelism, along with their hybrid compositions.”

“The PipeDream-1F1B strategy is the most widely adopted approach for PP [17].”

“In this way, the training pressure on a single node can be reduced, and the model transfer efficiency can be improved [27].”

“Deep analysis reveals that the intrinsic reason for this result is the ill-suited scheduling for PP. Current PP scheduling methods focus solely on computation to determine the execution order of microbatches on each stage [5, 8, 17] (referred to as computation operations hereafter).”

“To address this, later techniques such as 1F1B scheduling [24, 46] enhance memory utilization and minimize idle time by interleaving one forward and one backward pass per micro-batch at each training step.”

“To address this limitation, model parallelism (Harlap et al., 2018; Huang et al., 2019; Shoeybi et al., 2019; Zheng et al., 2022) partitions the model across multiple devices, making efficient multi-device training a central challenge.”

“Pipelined and resilient approaches demonstrate faster, memory-aware DNN learning across device networks [9]–[11], while recent surveys and healthcare deployments confirm their value in wireless and privacy-sensitive environments [12], [13].”

“Despite its effectiveness, this method introduces a significant communication bottleneck [Harlap et al., 2018] due to frequent synchronization.”

“They are an alternative to the 3D parallelism (Data [19], Model [25], and Pipeline [13]) that is usually used when training models across multiple-nodes with multiple GPUs.”

“Inter-operator parallelism, including pipeline parallelism (Harlap et al., 2018; Huang et al., 2019; Liu ZM et al., 2023), partitions models into several stages with multiple operators.”

“To address this problem, parallelism strategies, such as pipeline parallelism (PP) [10], [11] and data parallelism (DP) [12], [13], have been developed to distribute the models across multiple GPUs and to increase the throughput for inference and training.”

“PipeDream-2BW takes PipeDream-1F1B further by dividing the pipeline into two buckets to improve the overlap of computation and communication.”

“Model-parallelism systems [19, 44] pipeline the DNN model in heterogeneous clusters, which is near optimal for resource utilization with fine-tuning.”

“, pipeline parallel (PP) [3], [4], data parallel (DP) [5], [6], and operator parallel (OP) [7], [8], [9], specifically designed for the system and networks.”

“The point to note here is that each head has independent input, and thus all heads in MHAE can be executed in parallel and independently with each other by model parallelism [24].”

“Generally, models today have tens to hundreds of layers totaling tens of millions of parameters [2].”

“So, we use weight stashing introduced in [2] to alleviate this problem by storing the intermediate states of each batch to keep the same version of weights while update.”

“[7] implement PipeDream, a DNN training system for GPUs that parallelizes computation by pipelin-”

“…pipeline parallelism: with GPipe (PPgpipe) (Huang et al., 2018), the entire forward pass is run first, followed by the backward pass (Figure 4a), while with 1F1B (PP1f1b) (Harlap et al., 2018), the forward and backward steps are alternated so that earlier micro-batches finish as soon as possible.”

“Large-scale Trans-former training is enabled by tensor parallelism, pipeline parallelism and data parallelism [15], [16], [32], [33].”

“Pipeline parallel approaches such as PipeDream replace frequent global synchronization with stage-wise communication, while improved schedules reduce pipeline inefficiencies [24, 34].”

“In HPC settings, synchronous PP with micro-batching achieves high throughput and negligible communication overhead Harlap et al. (2018); Qi et al. (2023); Liu et al. (2024).”

“Distributed training methods, including tensor [13], [14] and pipeline parallelism [15], [11], often suffer from low hardware utilization, as GPUs and CPU cores remain idle during synchronization, pipeline bubbles, or data stalls [20], [21].”

“…and hybrid parallelism; memory optimization; distributed-training surveys, systems, and practical tutorials [25, 27, 40, 12, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53] Explain, implement, or optimize techniques for partitioning data, parameters, layers, activations, and…”

“Another drawback of pipeline parallelism is the necessity to perform asynchronous gradient updates, which degrades predictive performance [23].”

“…two AIDCs interconnected by wavelength-division multiplexing (WDM) optical networks, achieving a training e ffi ciency of 99.41%, which demonstrated the feasibility of pipeline-parallelism-based cross-AIDC DMT. GPipe [15] and PipeDream [16] are two primary methods for achieving pipeline parallelism.”

“Researchers have been optimizing LLM serving recently [3, 16, 35, 38, 43, 45, 48, 93].”

“Many parallelism strategies have been proposed, such as DP (Data Parallelism), TP [38], PP (Pipeline Parallelism) [9, 12, 18, 27], SP (Sequence Parallelism) [19, 22, 28, 32] and EP [24].”

“We adopt the Forward-then-Backward (F-then-B) strategy in GPipe [33], instead of One-Forward-One-Backward (1F1B) [17, 23, 51], as F-then-B better fits the memory and time optimizations when the number of microbatches is small.”

“Beyond the current GPipe implementation, we are extending support to 1F1B [24] and other pipeline schedules.”

“Existing approaches typically attempt to mitigate this inconsistency through weight stashing or equivalent [8, 10, 19, 27] or prediction, [3, 6, 7, 1] trading off memory, computation, or system complexity.”

“PD instead uses inter-batch pipelining with a one-forward-one-backward schedule to keep stages active (Harlap et al., 2018; Narayanan et al., 2019).”

“Based on this abstraction, FlexFlow supports graph construction and graph-level optimization [8], and naturally expresses common distributed execution patterns such as data parallelism, tensor parallelism, and pipeline parallelism [3], [4], [8], [10].”

“While communication differs across data parallelism (periodic bursts) [7,9,35,38], model parallelism (stage-specific spikes) [10,11,35,38], and hybrid parallelism (multi-phase flows) [8,35,37,38], their temporal distribution, bandwidth demand, and phase characteristics are jointly determined by the…”

“This can be done through Model Parallelism (MP) [14, 54], which divides the model into shards distributed across devices.”

“While asynchronous methods [27], [28] reduce flush-phase bubbles, they incur slower convergence and higher memory overhead from maintaining multiple weight versions.”

“[6,7] Its main overhead is the pipeline bubble, which shrinks as the micro-batch count M increases.”

“Instead of 3D parallelism, DLR OVER -LM employs a hybrid parallel strategy that integrates Fully Sharded Data Parallel (FSDP) [24] with Expert Parallelism (EP) [20], [25].”

“For work on CPU-GPU optimization in the context of distributed computing, we refer the reader to [7, 9, 17, 21].”

“For work on CPU-GPU optimization in the context of distributed computing, we refer the reader to [6, 7, 12, 14].”

“Prior work has explored data parallelism [6], tensor parallelism [7], and pipeline parallelism [8,9] as representative distributed training techniques for scaling both memory and computation.”

“[15] We typically used Tensor Parallelism or Pipeline Parallelism on two 910B cards to train the 7B models efficiently.”

“For instance, after applying basic asynchronous optimizations to PP, Pipedream [10] achieved over 40% higher training efficiency compared to synchronous pipeline approaches.”

“Thus, parallelism techniques have been explored like pipeline parallelism [14, 17] and model parallelism [24] which split models across multiple devices to take advantage of combined memory and compute overlapping.”

“Current planning algorithms lack robustness against such dynamics, leading to costly retraining [17], [18], [19].”

“However, as the number of worker nodes scales, the communication overhead associated with the all-reduce operation often becomes the primary bottleneck, consuming upwards of 90% of the per-iteration training time and significantly limiting overall throughput and scalability [7]–[9].”

“The classic 3D parallelism integrates data parallelism [1, 3], pipeline parallelism [10, 12, 22], and tensor Figure 1 Illustration of Asynchronous Hierarchical Zero Parallelism.”

“This introduces pipeline bubbles where some GPUs (which process former microbatches of a batch) need to wait for the weight updates from the other GPUs (which process latter microbatches of the batch) delaying the entire training process [72].”

“Pipeline Parallelism [8, 12–14, 23, 27, 28] partitions a model’s layers across devices (stages) and processes a batch as a stream of smaller micro-batches, requiring only minimal Peer-to-Peer communication.”

“Algorithms are designed to examine communication patterns in pipeline parallelism where model layers are allocated among various workers, which necessitates intelligent placement decisions minimizing inter-stage communication latency but maximizing pipeline throughput [6].”

“Elastic and memory-adaptive training frameworks such as HetPipe [50], PipeDream [51], and adaptive microbatching [52] address variable memory availability, often in distributed or heterogeneous hardware settings.”

“Such an approach is often referred to as software pipelining (Rau & Glaeser, 1981) and also applied to distributed systems (Harlap et al., 2018) (Huang et al., 2019).”

“In parallel, ColocRL [48], Mesh-Tensorflow [58], GPipe [26], PipeDream [22, 49], Tofu [69], GSPMD [77], TeraPipe [41], and GraphPipe [27] develop model parallelism for storing model parameters distributedly, while depending on inter-model communication; FlexFlow [44], Megatron-LM [59], DeepSpeed…”

“Most DL architectures are dominated by high-intensity matrix operators, and standard tooling for DL training optimization has evolved to support models that fit this pattern [15, 18, 20, 27, 39, 54, 56].”

“In a pipeline parallelism, each piece of data moves from stage to stage, eventually producing a final result (Harlap, et al., 2018; Huang, et al., 2019; Mastoras & Gross, 2018; Narayanan, et al., 2019; Yang P. , Zhang, Zhang, Yang, & Wei, 2022).”

“As indicated by Harlap et al. (2018) the MP is useful when working with large-scale models that do not fully fit into the memory of a single accelerator.”

“Asynchronous algorithms do not preform this flush, often resulting in a smaller bubble ratio at the cost of not converging as well as synchronous algorithms [11, 34, 35].”

“…network training, methodologies such as data parallelism (Krizhevsky et al., 2012), model parallelism (Dean et al., 2012), and pipeline parallelism (Harlap et al., 2018; Huang et al., 2019), have been adeptly applied to reduce the computational time for each evaluation of the loss function by…”

“That can be solved by tensor parallelism [6], [23]–[25] and pipeline parallelism [7], [26] where parameters are split in vertical and horizontal directions, respectively, but samples are the same for all devices.”

“In the context of DNN training, two prevalent strategies are model parallelism and data parallelism [11], [13], [22], [32], [33].”

“Indeed, these passes come with data dependencies such that, even if the model is partitioned, there will still be sequential operations which would make some of the computing resources idle [18], [19].”

“Harlap et al. also proposed a distributed pipelined system for DNN training [11].”

“Some researches extend decentralized SGD with a pipe (Harlap et al. 2018; Li et al. 2018) and perform well in ONE server equipped with several GPU cards.”

“Pipeline parallelism [10, 11] is to partition the whole model by layer in a serial manner, so that the input batch is processed on one device at a time, and then sent to the next device.”

“We compare our pipelined training scheme with two key existing systems: PipeDream (Harlap et al., 2018) and GPipe (Huang et al., 2018).”

“Adaptive batching extends this by automatically tuning batch sizes based on real-time workload conditions, balancing latency, throughput, and energy consumption [15].”

“The results in Table 5 incorporate DeepSpeed ZeRO-3 memory optimization (Rajbhandari et al., 2020), Liger Kernel (Hsu et al., 2025), pipeline parallelism(Harlap et al., 2018), and PCIe inter-connects under same compute node and same data scales each stage in the experiment before.”

“To solve the above problems, partitioning and pipelining strategies have been applied to distributed model parallelism to reduce partitioning issues and improve resource utilisation [3].”