Distributed LLM Serving on Consumer-Grade GPUs by Reconciling Computation and Communication

Petals focuses only on model placement and request scheduling.

“They are only focus on the model placement and request scheduling.”

• cites [4] [4] Alexander Borzunov, Dmitry Baranchuk, Tim Dettmers, Max Ryabinin, Younes Belkada, Artem Chumachenko, Pavel Samygin, and Colin Raffel. 2022. Petals: Collaborative inference and fine-tuning of large models. arXiv preprint arXiv:2209.01188. sourceDisputed

  The Introduction describes PETALS as a platform for collaborative inference and fine-tuning, and says trained submodules can be shared on a model hub. Section 2 likewise identifies inference, adaptation, and adapter sharing as use cases. Model placement and request scheduling are therefore not its only focus. (LLM, conf=0.91)

  Checked by a reasoning model — may be wrong.

  Experimental — accuracy not yet measured

Existing distributed serving systems suffer from transmission competition between prefill and decode requests.

“However, existing systems [28, 14, 4, 12] suffer from transmission competition between requests at different phases.”

• cites [28] [28] Gyeong-In Yu, Joo Seong Jeong, Geon-Woo Kim, Soojeong Kim, and Byung-Gon Chun. 2022. Orca: A distributed serving system for {Transformer-Based} generative models. In 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22), pages 521–538.Not checked yet

  Not checked in the free run.

• cites [14] [14] Yixuan Mei, Yonghao Zhuang, Xupeng Miao, Juncheng Yang, Zhihao Jia, and Rashmi Vinayak. 2025. Helix: Serving large language models over heterogeneous gpus and network via max-flow. AS- PLOS '25. Association for Computing Machinery. sourceCould not verify

  The passages from our copy are repository file listings, with no section or page labels. Searches for transmission competition between phases and for prefill–decode network contention found no passage addressing whether existing systems suffer from it. (LLM, conf=0.72)

  Experimental — accuracy not yet measured
• cites [4] [4] Alexander Borzunov, Dmitry Baranchuk, Tim Dettmers, Max Ryabinin, Younes Belkada, Artem Chumachenko, Pavel Samygin, and Colin Raffel. 2022. Petals: Collaborative inference and fine-tuning of large models. arXiv preprint arXiv:2209.01188. sourcePartly supported

  Section 3.3 (Benchmarks; page unavailable) reports that eight clients running inference concurrently each experienced about a 20% slowdown, supporting a general contention concern. It does not attribute that slowdown to transmission competition or distinguish prefill from decode requests. (LLM, conf=0.76)

  Experimental — accuracy not yet measured
• cites [12] [12] Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gon- zalez, Hao Zhang, and Ion Stoica. 2023. Efficient memory management for large language model serv- ing with pagedattention. In Proceedings of the 29th Symposium on Operating Systems Principles, pages 611–626. sourceCould not verify

  Section 2.2 describes the prompt phase, and §4.5 discusses scheduling when requests exceed capacity. Neither passage says that prefill and decode requests compete for transmission resources. Searches for transmission contention and competition between phases found no passage addressing that claim. (LLM, conf=0.72)

  Experimental — accuracy not yet measured

Existing systems often set the number of micro-batches equal to the degree of pipeline parallelism.

“existing systems often set the number of micro-batches equal to the degree of pipeline parallelism [12, 28].”

• cites [12] [12] Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gon- zalez, Hao Zhang, and Ion Stoica. 2023. Efficient memory management for large language model serv- ing with pagedattention. In Proceedings of the 29th Symposium on Operating Systems Principles, pages 611–626. sourceCould not verify

  Section 2.3, “Batching Techniques for LLMs” (page unavailable), discusses batching requests and iteration-level scheduling, but not micro-batches in a pipeline. Searches for micro-batches, pipeline parallelism, and pipeline stages found no passage stating that systems often set the micro-batch count equal to the degree of pipeline parallelism. (LLM, conf=0.73)

  Experimental — accuracy not yet measured
• cites [28] [28] Gyeong-In Yu, Joo Seong Jeong, Geon-Woo Kim, Soojeong Kim, and Byung-Gon Chun. 2022. Orca: A distributed serving system for {Transformer-Based} generative models. In 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22), pages 521–538.Not checked yet

  Not checked in the free run.

Prefill requests can involve transmitting up to 1000 times more data than decode requests.

“Prefill requests can involve transmitting up to 1000 times more data than decode requests, as their volume scales with the number of processed tokens.”

Prefill is communication bound when the number of tokens exceeds the compute saturation point (256).

“On the other hand, although the number of tokens for prefill can easily exceed the compute saturation point t (i.e., 256), the communication overhead is much greater than the computation overhead, which we refer to as a communication bound task.”

When both chunk transmission and micro-batch extension are combined, they reduce TPOT by 14.6%.

“When both techniques are combined, they reduce TPOT by 14.6%”

Chunk transmission reduces TPOT by 8.4%.

“chunk transmission reduces TPOT by 8.4%”

MoLink achieves the most reduction in TTFT, TPOT, and end-to-end latency compared to vLLM and Helix at an arrival rate of 0.3 req/s.

“This is the point where MoLink achieves the most reduction in TTFT, TPOT, and end-to-end latency compared to vLLM and Helix.”

In real practice, the delay of decode can be hundreds of times greater compared to transmission without competition.

“In real practice, we observe that the delay of decode can be hundreds of times greater compared to the transmission without competition.”

Transmission competition between prefill and decode occurs when sequentially processing a new arriving prefill request and an existing batch of decode requests.

“The competition often happens when sequentially processing a new arriving request in prefill phase and an existing batch of requests in decode phase.”

MoLink benefits from the proposed optimization techniques across a wide range of bandwidth.

“We can see that MoLink benefits from the proposed optimization techniques across a wide range of bandwidth.”

Micro-batch extension reduces TPOT by 10.5%.

“micro-batch extension reduces TPOT by 10.5%”

When servers are connected via limited bandwidth, transmission overhead for activations can increase significantly.

“However, when servers are connected via limited bandwidth, the transmission overhead for activations can increase significantly.”

Increased transmission overhead can lead to pipeline bubbles, reducing overall system efficiency.

“This can lead to pipeline bubbles, reducing the overall efficiency of the system.”

The decrease in MoLink's benefits as delay increases is because pipeline bubbles become more frequent due to more delayed arrival of micro-batches.

“This is because pipeline bubbles become more frequent due to the more delayed arrival of micro-batches, as illustrated in Figure 6.”

The optimal number of micro-batches can differ according to hardware and network conditions.

“The optimal number of micro-batch can be different according to both the hardware and network conditions.”

The server receiving the decode request remains idle while the decode request is delayed by prefill transmission.

“Consequently, the server receiving the decode request remains idle during this period.”

• Also stated in: § 3.2 Transmission competition of prefill and · plain

  “during which the transmission of decode is blocked and the server2 idles at the time between t1 and t2.”