Distributed Inference and Fine-tuning of Large Language Models Over The Internet

If none of the servers fail, the fault-tolerant algorithm recovers O(n • t) communication, similarly to Aminabadi et al. (2022).

“If none of the servers fail, we recover O(n • t) communication, similarly to Aminabadi et al. (2022).”

• cites Aminabadi et al. (2022) sourceCould not verify

  The PCC section discusses GPU all-to-all communication latency, giving bounds of O(p/L) and O(p/L) + O(L). Searches for fault tolerance, server failures, and O(n • t) communication found no passage addressing the claimed failure-free case. (LLM, conf=0.76)

  Experimental — accuracy not yet measured

• AssumptionThe authors state this as an assumption: Every block is hosted on several servers.

Our system significantly outperforms offloading even when each GPU node runs its own client doing single-batch inference at the same time.

“Crucially, our system significantly outperforms offloading even when each GPU node runs its own client doing single-batch inference at the same time.”

Inference is more challenging than fine-tuning for cost-efficient setups.

“Counter-intuitively, we found that inference is more challenging than fine-tuning for cost-efficient setups.”

With realistic network speeds, the distributed algorithms perform autoregressive generation ≥10× faster than local offloading.

“With realistic network speeds, our distributed algorithms perform autoregressive generation ≥10× faster than local offloading.”

Fine-tuning forward passes for large batches are affected by both bandwidth and latency.

“In turn, fine-tuning forward passes for large batches are affected by both bandwidth and latency.”

Recomputing all previous tokens on every inference step requires O(t^3) time for transformer-based models.

“Naturally, this approach needs increasingly more computation with sequence length t, for a total of O(t 3 ) time for transformer-based modelsfoot_3 .”

Letting pipeline stages communicate directly reduces the total step time.

“This reduces the total step time since both messages are a few kilobytes in size an can be sent in parallel.”

The real-world setup turns out to be slower than the A100 benchmarks due to slower hardware.

“Finally, the real-world setup turns out to be slower than the A100 benchmarks due to slower hardware.”

PETALS runs Llama 2 (70B) and BLOOM (176B) over the Internet up to 10× faster than offloading for interactive generation.

“We showcase these algorithms in PETALS 1 -a decentralized system that runs Llama 2 (70B) and BLOOM (176B) over the Internet up to 10× faster than offloading for interactive generation.”

Our algorithm outperforms offloading even when communicating between different continents.

“Still, our algorithm outperforms offloading even when communicating between different continents.”

The fault-tolerant algorithm provides reasonable performance in all tested conditions, especially for higher failure rates.

“Unlike baselines, our algorithm provides reasonable performance in all tested conditions, especially for higher failure rates (common for communicating over the Internet, using spot/preemptible instances or unreliable hardware).”

• AssumptionThe authors state this as an assumption: Every block is hosted on several servers.

Existing parameter offloading techniques are inefficient in many use cases, such as LLM-based chatbots and search engines.

“However, as we show in Section 3.1, these techniques are inefficient in many use cases, such as LLM-based chatbots and search engines.”

Parameter offloading can still be efficient when generating large amounts of short sequences in bulk.

“Parameter offloading can still be efficient when generating large amounts of short sequences in bulk.”

Caching half-precision activations of a 2048-token sequence for large models like GPT-3 or OPT-175B takes up 9.6 GB GPU memory for each sequence.

“Caching half-precision activations of a 2048-token sequence for large models like GPT-3 (Brown et al., 2020) or OPT-175B (Zhang et al., 2022) (with 96 layers of 12288 units each) takes up 9.6 GB GPU memory for each sequence.”

A large enough model (50B+) can run efficiently even on geodistributed devices in a consumer-grade network.

“We observe that a large enough model (50B+) can run efficiently even on geodistributed devices in a consumer-grade network.”

In other cases, our algorithm offers higher throughput than offloading for training.

“In other cases, our algorithm offers higher throughput than offloading for training.”

For models with over 50B parameters, communicating activations over a slow network can be faster than swapping layers from local RAM or SSD.

“Notably, for models with over 50B parameters, communicating activations over a slow network can be faster than swapping layers from local RAM or SSD.”

Recomputing all previous tokens on every inference step is often more efficient than offloaded caching, especially for shorter sequences.

“Surprisingly, this approach is often more efficient than offloaded caching, especially for shorter sequences due to the overhead from loading and storing cache from RAM or SSD.”

For parallel forward passes, offloading is competitive if networking is limited to 100 Mbit/s or has high latency.

“For parallel forward passes, offloading is competitive if networking is limited to 100 Mbit/s or has high latency.”

The proposed system significantly outperforms other approaches to running inference on consumer-grade hardware.

“which significantly outperforms other approaches to running inference on consumer-grade hardware.”

The throughput of the GPU-enabled client is similar (within 10% difference) to the throughput in sequence classification.

“We observe that the throughput of the GPU-enabled client is similar (within 10% difference) to the throughput in case of sequence classification, reported in Table 6.”

The rebalancing algorithm does not cause oscillations because block replacements are executed only if they eventually increase throughput by at least p%.

“The rebalancing algorithm does not cause oscillations since a series of block replacements is executed only if it leads to eventually increasing throughput by at least p%.”

• Also stated in: § Stability of the greedy algorithm. · boosted

  “We verified this behavior computationally, simulating a network with thousands of servers with different throughputs.”

Prefix tuning is slower than prompt tuning because it adds several times more trainable parameters.

“The prefix tuning turns out to be slower, since it adds several times more trainable parameters.”

The baseline running load balancing for new servers only achieves reasonable throughput during periods where servers are actively joining.

“Curiously, the baseline running load balancing for new servers only achieves reasonable throughput during periods where servers are actively joining.”

The naive baseline with random layer assignment has zero throughput most of the time because it is unable to form a complete pipeline.

“Finally, the naive baseline with random layer assignment has zero throughput most of the time because it is unable to form a complete pipeline.”

The greedy block assignment algorithm usually finds an assignment with total throughput of 90-100% of the optimal one, given realistic throughput values.

“we have conducted computational experiments and found out that this greedy algorithm (running in polynomial time) usually finds an assignment with total throughput of 90-100% of the optimal one (found by trying out all possible assignments in exponential time), given that the values of throughput are realistic to our setup.”

• AssumptionThe authors state this as an assumption: The throughput values used in the block assignment experiments are realistic to the setup.

The proposed system can scale to the largest publicly available language model with hundreds of billions of trainable parameters.

“We demonstrated that the proposed system can scale to the largest publicly available language model with hundreds of billions of trainable parameters.”

Servers can simultaneously run different fine-tuning tasks without them interfering with one another.

“Thus, a server can simultaneously run different fine-tuning tasks without them interfering with one another.”

The decentralized load-balancing algorithm assigns transformer blocks to every server to maximize the total system throughput.

“Finally, to make sure that there are enough servers for every part of the model, we develop a decentralzied load-balancing algorithm that assigns transformer blocks to every server to maximize the total system throughput.”

• AssumptionThe authors state this as an assumption: Each server holds a segment of consecutive transformer blocks to minimize inference latency.

PETALS allows users to run inference and fine-tuning over a swarm of unreliable devices with the same correctness guarantees as when running locally.

“The system allows users to run inference and fine-tuning over a swarm of unreliable devices with the same correctness guarantees as when running locally.”

• AssumptionThe authors state this as an assumption: Every block is hosted on several servers.

The next joining server would always cover a block with the smallest t_i.

“This way, the next joining server would always cover a block with the smallest t i .”

• AssumptionThe authors state this as an assumption: Each server holds a segment of consecutive transformer blocks to minimize inference latency.

Given the same hardware, a group of researchers will get much better inference speed by collaborating over the Internet using our system compared to each of them running offloading independently.

“Thus, given the same hardware, a group of researchers will get much better inference speed by collaborating over the Internet using our system compared to each of them running offloading independently.”

If all servers fail after one step, the fault-tolerant algorithm effectively performs non-caching generation, which is the best option in that scenario.

“In turn, if all servers fail after one step, the algorithm effectively performs non-caching generation, which is the best option in that scenario.”

• AssumptionThe authors state this as an assumption: Every block is hosted on several servers.

Parameter offloading is incompatible with in-context learning or prompt engineering, where the model needs to process long sequences of training examples.

“For instance, it is incompatible with in-context learning or prompt engineering, where the model needs to process long sequences of training examples (Brown et al., 2020).”

The fault-tolerant distributed autoregressive inference algorithm using dual attention caches can quickly recover from a failed server and reassign the load to one or more replacement servers.

“Using dual attention caches, this algorithm can quickly recover from a failed server and reassign the load to one or more replacement servers.”

• AssumptionThe authors state this as an assumption: Every block is hosted on several servers.

Even consumer-grade GPUs like GeForce RTX 3070 could run a complete inference step of BLOOM-176B in less than a second in terms of raw FLOPs.

“In terms of raw FLOPs, even consumer-grade GPUs like GeForce RTX 3070 could run a complete inference step of BLOOM-176B in less than a second (NVIDIA, 2020).”

• cites NVIDIA, 2020Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Quantization reduces the number of consecutive devices and communication rounds.

“To make this more efficient, we use quantization to store more parameters per GPU, reducing the number of consecutive devices and communication rounds.”

Our system may experience an imbalance between supply and demand because people using the client are not required to run a server.

“Since people using the client are not required to run a server, our system may experience an imbalance between supply (peers who dedicate GPUs to serve model layers) and demand (peers using the servers to perform inference or fine-tuning for their own needs).”

The heaviest operation the client does in sequence classification is multiplying 2 × h and h × b matrices.

“In case of sequence classification, the heaviest operation the client does is multiplying 2 × h and h × b matrices, where h is the hidden dimension (14336 in BLOOM-176B) and b is the batch size.”