Petals: Collaborative Inference and Fine-tuning of Large Models

Offloading allows running LLMs with a single low-end accelerator by loading parameters from RAM just-in-time for each forward pass.

“This method allows running LLMs with a single low-end accelerator by loading parameters from RAM justin-time for each forward pass.”

• cites Pudipeddi et al., 2020 sourcePartly supported

  The Abstract (p. 1) and §1.2 (p. 3) describe keeping the model in host DRAM and transferring layers to a single device as needed. But the demonstrated device is a 16 GB V100, not a low-end accelerator, and L2L runs multiple micro-batches on each layer rather than loading parameters for each forward pass. (LLM, conf=0.86)

  Experimental — accuracy not yet measured
• cites Ren et al., 2021 sourceDisputed

  Section 4.1 (Single GPU Schedule) says ZeRO-Offload keeps fp16 parameters on the GPU, so forward propagation requires no CPU communication. Although it enables large-model training on a single GPU, it does not load parameters from RAM just in time for each forward pass; the paper describes that layer-by-layer transfer as L2L's approach in Section 2. (LLM, conf=0.90)

  Checked by a reasoning model — may be wrong.

  Experimental — accuracy not yet measured

Parameter-efficient approaches are often competitive with full model fine-tuning.

“Despite their lower memory requirements, parameter-efficient approaches are often competitive with full model fine-tuning (Hu et al., 2021;Liu et al., 2021a;Yong and Nikoulina, 2022) and even outperform it in low-data regimes (Liu et al., 2022b).”

• cites Hu et al., 2021Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Liu et al., 2021aNot matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Yong and Nikoulina, 2022 sourcePartly supported

  The Results and Discussion (p. 4) say that lightweight adaptation nearly closes the gap with upper-bound baselines for German, supporting competitiveness in that setting. The gap remains substantial for Korean, and the paper does not directly compare these approaches with full fine-tuning of the same model, so it does not establish the broader claim. (LLM, conf=0.82)

  Experimental — accuracy not yet measured

LLMs often need training to achieve the best results.

“While LLMs achieve high quality on many problems with simple prompt engineering (Brown et al., 2020), they often need training to achieve the best results.”

• cites Brown et al., 2020 sourceSupported

Quantization has about 5% overhead with batch size 1 (20 tokens).

“In terms of inference time, Table 2 demonstrates that quantization has about 5% of overhead with batch size 1 (20 tokens), but becomes negligible for larger batches.”

8-bit mixed matrix decomposition has little effect on LLM quality for major benchmarks.

“As shown in Table 1, this method has little effect on LLM quality for major benchmarks.”

Offloading is about an order of magnitude slower for single-batch inference compared to PETALS.

“We can see that offloading is about an order of magnitude slower for single-batch inference compared to PETALS.”

PETALS outperforms offloading for very large models.

“We demonstrate that this strategy outperforms offloading for very large models, running inference of BLOOM-176B on consumer GPUs with ≈ 1 step per second, which is enough for many interactive LLM applications.”

In other cases, PETALS offers higher throughput than offloading for training.

“In other cases, PETALS offers higher throughput than offloading for training.”

Quantization overhead becomes negligible for larger batches.

“In terms of inference time, Table 2 demonstrates that quantization has about 5% of overhead with batch size 1 (20 tokens), but becomes negligible for larger batches.”

Parallel forward passes with large batches are affected by both bandwidth and latency.

“Parallel forward passes with large batches (used for fine-tuning and parallel inference) are affected by both bandwidth and latency.”

In PETALS, the intermediate activations of the model for given inputs are sent to the servers without any encryption.

“An important limitation of our work is data privacy: the intermediate activations of the model for given inputs are sent to the servers without any encryption.”

PETALS is a system for efficient collaborative inference and fine-tuning of large language models.

“This paper introduces PETALS, a system for efficient collaborative inference and fine-tuning of large language models.”

For 12 servers with 100 Mbit/s bandwidth and 100 ms latency, if 8 clients run inference concurrently, each gets approximately 20% slowdown compared to running alone.

“For 12 servers with 100 Mbit/s bandwidth and 100 ms latency, if 8 clients run inference concurrently, each of them gets ≈ 20% slowdown compared to the case when it runs inference alone.”

Compressing weights to 8-bit precision changes the number of required nodes for BLOOM from 44 to 22.

“For BLOOM, this changes the number of required nodes from 44 to 22, which reduces latency in half and decreases the probability of a failure.”

• AssumptionThe authors state this as an assumption: Each server has at least 16 GB of CPU RAM and 8 GB of GPU memory.

PETALS develops algorithms for reliable routing and load balancing.

“In addition, we develop algorithms for reliable routing and load balancing.”

Unlike most inference APIs, PETALS natively exposes hidden states of served models.

“Unlike most inference APIs, PETALS also natively exposes hidden states of served models, allowing to train and share custom model extensions based on efficient fine-tuning methods.”

PETALS runs inference of BLOOM-176B on consumer GPUs at approximately 1 step per second.

“We demonstrate that this strategy outperforms offloading for very large models, running inference of BLOOM-176B on consumer GPUs with ≈ 1 step per second, which is enough for many interactive LLM applications.”

Offloading has inherently high latency.

“Offloading can be efficient for processing many tokens in parallel, but it has inherently high latency: for example, generating one token at a time with BLOOM-176B takes at least 5.5 seconds for the fastest RAM offloading setup and 22 seconds for the fastest SSD offloading.”

The more participants in generation, the higher the chance that one fails and the entire procedure needs to restart.

“Also, the more participants take part in the generation procedure, the higher the chance that one of them fails and the entire procedure needs to restart.”

A naive restart solution for server failures has scaling issues: longer sequences require more frequent restarts, increasing inference time superlinearly.

“A naive solution would be to restart the generation procedure, treating previously generated tokens as part of the prompt.This approach has two scaling issues.When generating longer sequences, the inference would have to restart more often, increasing the inference time superlinearly.”

PETALS interface can support other popular parameter-efficient fine-tuning algorithms, such as LoRA or prefix tuning.

“This interface can also support other popular parameter-efficient fine-tuning algorithms, such as LoRA (Hu et al., 2021) or prefix tuning (Li and Liang, 2021).”

• cites Hu et al., 2021Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Li and Liang, 2021Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

In the worst case, BLOOM is distributed among 44 nodes (352 GB / 8 GB per server).

“Thus, in the worst case, the model is distributed among 352 GB / 8 GB (per server) = 44 nodes.”

• AssumptionThe authors state this as an assumption: Each server has at least 16 GB of CPU RAM and 8 GB of GPU memory.

The reduction from 44 to 22 nodes decreases the probability of a failure.

“For BLOOM, this changes the number of required nodes from 44 to 22, which reduces latency in half and decreases the probability of a failure.”

• AssumptionThe authors state this as an assumption: Each server has at least 16 GB of CPU RAM and 8 GB of GPU memory.

Current API pricing can make some research projects prohibitively expensive.

“On top of that, current API pricing can make some research projects prohibitively expensive (Liu et al., 2022a).”

• cites Liu et al., 2022aNot matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

If all peers serving certain blocks leave, the rebalancing procedure quickly redistributes remaining resources to close gaps.

“In particular, if all peers serving certain blocks suddenly leave the system, this procedure quickly redistributes the remaining resources to close the emerged gaps.”

Parameter-efficient approaches even outperform full model fine-tuning in low-data regimes.

“Despite their lower memory requirements, parameter-efficient approaches are often competitive with full model fine-tuning (Hu et al., 2021;Liu et al., 2021a;Yong and Nikoulina, 2022) and even outperform it in low-data regimes (Liu et al., 2022b).”

• Also stated in: § Security. · plain

  “and even outperform it in low-data regimes(Liu et al., 2022b).”

• cites Liu et al., 2022bNot matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

PETALS's inference speed of approximately 1 step per second for BLOOM-176B is enough for many interactive LLM applications.

“We demonstrate that this strategy outperforms offloading for very large models, running inference of BLOOM-176B on consumer GPUs with ≈ 1 step per second, which is enough for many interactive LLM applications.”

OPT-175B and BLOOM-176B need significantly more than 350 GB accelerator memory for fine-tuning.

“For instance, OPT-175B and BLOOM-176B need over 350 GB accelerator memory for inference and significantly more for fine-tuning.”

For very large models, fine-tuning all model parameters becomes impractical due to hardware requirements.

“However, for very large models, this strategy becomes impractical due to hardware requirements.”

During generation, inference time is mostly sensitive to network latency.

“During generation, clients process one or few tokens at a time; in practice, the inference time is mostly sensitive to the network latency.”

A key limitation of PETALS is that peers serving the first layers of the model can use their inputs to recover input tokens.

“A key limitation of our approach is that peers serving the first layers of the model can use their inputs to recover input tokens.”

Parameter-efficient approaches are often competitive with full model fine-tuning despite their lower memory requirements.

“Despite their lower memory requirements, parameter-efficient approaches are often competitive with full model fine-tuning(Hu et al., 2021; Liu et al., 2021a;Yong and Nikoulina, 2022)”

• cites Hu et al., 2021; Liu et al., 2021a;Yong and Nikoulina, 2022Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

The reduction from 44 to 22 nodes reduces latency in half.

“For BLOOM, this changes the number of required nodes from 44 to 22, which reduces latency in half and decreases the probability of a failure.”

• AssumptionThe authors state this as an assumption: Each server has at least 16 GB of CPU RAM and 8 GB of GPU memory.

Making intermediate servers send output activations directly to the next server reduces the next server's start time to one network hop instead of two.

“To address this, we can make intermediate servers send the output activations both (a) directly to the next server and (b) back to the client.This way, the next server will start computations as soon as possible (after only one network hop instead of two hops), while the client will still be able to reuse the activations in case of server failures.”

PETALS allows training and sharing custom model extensions based on efficient fine-tuning methods.

“Unlike most inference APIs, PETALS also natively exposes hidden states of served models, allowing to train and share custom model extensions based on efficient fine-tuning methods.”

The algorithm where clients send requests and receive responses from servers one by one is suboptimal for sequential inference, where performance is bounded by network latency.

“The algorithm above implies that clients send requests and receive responses from servers one by one, while servers do not directly pass activations to each other.This is suboptimal for sequential inference, where performance is bounded by the network latency.”

It might be possible for people hosting the servers to recover the user's input data from the intermediate activations.

“As such, it might be possible for people hosting the servers to recover the user's input data.”

Generating one token at a time with BLOOM-176B takes 22 seconds for the fastest SSD offloading.

“Offloading can be efficient for processing many tokens in parallel, but it has inherently high latency: for example, generating one token at a time with BLOOM-176B takes at least 5.5 seconds for the fastest RAM offloading setup and 22 seconds for the fastest SSD offloading.”

BLOOM has 176B parameters, which takes 352 GB of GPU memory in 16-bit precision.

“For example, BLOOM has 176B parameters, which takes 352 GB of GPU memory in 16-bit precision.”

Generating one token at a time with BLOOM-176B takes at least 5.5 seconds for the fastest RAM offloading setup.

“Offloading can be efficient for processing many tokens in parallel, but it has inherently high latency: for example, generating one token at a time with BLOOM-176B takes at least 5.5 seconds for the fastest RAM offloading setup and 22 seconds for the fastest SSD offloading.”

These LLMs usually require multiple high-end GPUs or multi-node clusters to be run.

“As a result, these LLMs usually require multiple highend GPUs or multi-node clusters to be run.”

As more and more users train the base model, it will effectively become more capable over time.

“This capability provides a meaningful step towards collaborative improvement of machine learning models (Raffel, 2021): as more and more users train the base model, it will effectively become more capable over time.”

• cites Raffel, 2021Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

In terms of raw FLOPs, even consumer-grade GPUs like GeForce RTX 3070 could run a complete inference step of BLOOM-176B in less than a second.

“In terms of raw FLOPs, even consumer-grade GPUs like GeForce RTX 3070 could run a complete inference step of BLOOM-176B in less than a second (NVIDIA, 2020).”

• cites NVIDIA, 2020Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Servers maximize total model throughput by choosing blocks with the lowest throughput, eliminating bottlenecks.

“Formally, servers maximize the total model throughput by choosing the blocks with the lowest throughput, thus eliminating potential bottlenecks.”

Clients can simultaneously run different training tasks on the same set of servers without interfering with one another.

“Thus, clients can simultaneously run different training tasks on the same set of servers without interfering with one another.”