CoChat
Sign Up
PreprintCited by 8Topic ModelingNatural Language Processing TechniquesAdvanced Neural Network Applications

Petals: Collaborative Inference and Fine-tuning of Large Models

Open access
Borzunov, Alexander, Baranchuk, Dmitry, Dettmers, Tim, Ryabinin, Max, Belkada, Younes, Chumachenko, Artem, Samygin, Pavel, Raffel, Colin·2022·arXiv
DOI 10.48550/arxiv.2209.01188arXiv 2209.01188
Open this paper in CoChat — free accountOpen PDF

Abstract

Many NLP tasks benefit from using large language models (LLMs) that often have more than 100 billion parameters. With the release of BLOOM-176B and OPT-175B, everyone can download pretrained models of this scale. Still, using these models requires high-end hardware unavailable to many researchers. In some cases, LLMs can be used more affordably via RAM offloading or hosted APIs. However, these techniques have innate limitations: offloading is too slow for interactive inference, while APIs are not flexible enough for research that requires access to weights, attention or logits. In this work, we propose Petals - a system for inference and fine-tuning of large models collaboratively by joining the resources of multiple parties. We demonstrate that this strategy outperforms offloading for very large models, running inference of BLOOM-176B on consumer GPUs with $\approx$ 1 step per second, which is enough for many interactive LLM applications. Unlike most inference APIs, Petals also natively exposes hidden states of served models, allowing to train and share custom model extensions based on efficient fine-tuning methods.

AI snapshot

Written from the paper’s full text.

TL;DR

PETALS is a system for collaborative inference and parameter-efficient fine-tuning of large language models by chaining servers that each hold a subset of Transformer layers. It runs BLOOM-176B inference on consumer GPUs at roughly 1 step per second and uses 8-bit compression, routing, and load balancing to make 100B+ models more accessible.

Key findings

  • PETALS runs BLOOM-176B inference on consumer GPUs at ≈1 step per second, enough for many interactive LLM applications; in real-world tests on 14 servers across Europe and North America, single-batch inference reached 0.83 steps/s at sequence length 128 and 0.79 steps/s at sequence length 2048.
  • 8-bit weight quantization reduces the number of nodes needed to hold BLOOM from 44 to 22, halving latency, with little quality loss: zero-shot average was 75.3 (16-bit) vs. 74.9 (8-bit) for OPT-175B and 70.1 vs. 70.3 for BLOOM.
  • Parallel forward passes for fine-tuning/parallel inference reached 32.6 tokens/s at batch size 1 and 179.4 tokens/s at batch size 64 on the 14 real servers; offloading was about an order of magnitude slower for single-batch inference.
  • With 12 servers at 100 Mbit/s and 100 ms latency, 8 concurrent clients each saw ≈20% slowdown compared to running inference alone.

Methods

  • System design: servers host subsets of model layers and handle requests; clients form pipeline-parallel chains of consecutive servers, store embeddings locally, and use hivemind for decentralized fault tolerance, load balancing, and client-side routing.
  • Training: clients own trainable parameters while servers run backpropagation through their layers without updating server-side parameters; the interface supports parameter-efficient methods such as soft prompt tuning, adapters, LoRA, and prefix tuning.
  • Evaluation: BLOOM-176B was benchmarked in emulated setups (3 local A100 servers; 12 virtual servers) and a real-world setup of 14 servers with mixed consumer/prosumer GPUs across Europe and North America, under several bandwidth/latency configurations, and compared against parameter offloading upper bounds.

Limitations

  • Stated: privacy is limited because peers serving the first layers can use their inputs to recover input tokens; users with sensitive data should restrict clients to trusted servers or set up an isolated PETALS swarm.
  • Stated: security assumes independent server operators, but faulty or malicious servers may return incorrect outputs; the paper proposes pledge/hash-based economic mechanisms but does not fully solve this.
  • Stated: incentives may be imbalanced because clients are not required to run servers; the paper suggests points or rewards but leaves this as future work. Evident from the text: the evaluation focuses on BLOOM-176B system performance and quantization benchmarks, with no end-to-end downstream task results for distributed fine-tuning reported.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

DataCite · 10.48550/arxiv.2209.01188

Cites

0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

5 of 13 cited sources checked · 3 supported · 0 flagged · accuracy not yet measured

  • Findingown data
    § Abstract · p. 1

    PETALS outperforms offloading for very large models.

    “We demonstrate that this strategy outperforms offloading for very large models, running inference of BLOOM-176B on consumer GPUs with ≈ 1 step per second, which is enough for many interactive LLM applications.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Generating one token at a time with BLOOM-176B takes at least 5.5 seconds for the fastest RAM offloading setup and 22 seconds for the fastest SSD offloading.
  • Findingown data
    § 3.3 Benchmarks · p. 6

    In other cases, PETALS offers higher throughput than offloading for training.

    “In other cases, PETALS offers higher throughput than offloading for training.”
    Quote in the paperNot a substantial claim
  • Findingown data
    § 3.3 Benchmarks · p. 6

    With 12 servers at 100 Mbit/s bandwidth and 100 ms latency, if 8 clients run inference concurrently, each gets approximately 20% slowdown compared to running alone.

    “For 12 servers with 100 Mbit/s bandwidth and 100 ms latency, if 8 clients run inference concurrently, each of them gets ≈ 20% slowdown compared to the case when it runs inference alone.”
    Quote in the paperNot a substantial claim
  • Findingown data
    § 3.3 Benchmarks · p. 6

    Offloading is about an order of magnitude slower for single-batch inference compared to PETALS.

    “We can see that offloading is about an order of magnitude slower for single-batch inference compared to PETALS.”
    Quote in the paperSubstantial claim
  • Findingown data
    § 3.3 Benchmarks · p. 6

    For single-batch inference, performance degrades with higher latency.

    “degrades with higher latency”
    Quote in the paperSubstantial claim
  • May combine two claims. 8
  • Classified as probably not a substantial claim. 8
  • The quote only partly states this claim. 1

  • May combine two claims.

    Premisecitation
    § 3.1 Large model inference on consumer GPUs · p. 5

    Dynamic blockwise quantization halves the bandwidth requirements without any noticeable effect on generation quality.

    “Dynamic blockwise quantization halves the bandwidth requirements without any noticeable effect on generation quality.”
    Quote in the paperSubstantial claim1 of 2 sources checked
    • cites Dettmers et al., 2022b

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Ryabinin et al., 2023 source

      In Appendix I (p. 23), Table 7 reports that dynamic blockwise 8-bit quantization halved data transfer relative to the 16-bit baseline and produced similar final perplexity (21.13 versus 21.02). The paper evaluates perplexity, not generated-output quality, and the halving applies to activation data transfer rather than necessarily to total bandwidth requirements. (LLM, conf=0.85)

      Experimental — accuracy not yet measured

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § 1 Introduction · p. 1

    Modern language models often have hundreds of billions of parameters.

    “Following this trend, modern language models often have hundreds of billions of parameters (Brown et al., 2020; Rae et al., 2021; Zeng et al., 2021; Kim et al., 2021).”
    Quote in the paperNot a substantial claim3 of 3 sources checked
    • cites Brown et al., 2020 source

      The Conclusion (p. 40) describes a 175-billion-parameter language model, supporting the point that modern models can be very large. It does not establish that models often have hundreds of billions of parameters; the cited model is also below 200 billion. (LLM, conf=0.83)

      Experimental — accuracy not yet measured
    • cites Rae et al., 2021 source
    • cites Zeng et al., 2021 source

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § 1 Introduction · p. 1

    Performance tends to improve as language model scale increases.

    “Furthermore, performance tends to improve as scale increases (Radford et al., 2019; Kaplan et al., 2020).”
    Quote in the paperNot a substantial claim1 of 2 sources checked
    • cites Radford et al., 2019

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Kaplan et al., 2020

  • May combine two claims.

    Findingown data
    § 3.1 Large model inference on consumer GPUs · p. 5

    Quantization has about 5% overhead with batch size 1 (20 tokens), but becomes negligible for larger batches.

    “In terms of inference time, Table 2 demonstrates that quantization has about 5% of overhead with batch size 1 (20 tokens), but becomes negligible for larger batches.”
    Quote in the paperNot a substantial claim

  • The quote only partly states this claim.

    Findingown data
    § 3.3 Benchmarks · p. 6

    PETALS on 14 real servers in Europe and North America achieves 0.83 steps/s for single-batch inference with sequence length 128.

    “Real world | 0.83 | 0.79 | 32.6 | 179.4”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § 3.3 Benchmarks · p. 6

    For the fine-tuning forward pass, offloading is competitive if multiple GPUs are used and the networking for PETALS is limited to 100 Mbit/s or has high latency.

    “For the fine-tuning forward pass, offloading is competitive if multiple GPUs are used and the networking for PETALS is limited to 100 Mbit/s or has high latency.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Findingnone given
    § 5 Conclusion · p. 7

    8-bit compression reduces the resource requirements to run very large models.

    “We use 8-bit compression that reduces the resource requirements to run very large models.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning
    § 3.3 Benchmarks · p. 6

    Offloading 176B parameters takes 5.5 seconds for a regular setup and 11 seconds for a multi-GPU setup.

    “As such, offloading 176B parameters takes 5.5 seconds for a regular setup and 11 seconds for a multi-GPU setup.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: In 8-bit, the model uses 1 GB of memory per billion parameters.
    • Rests on an assumption: PCIe 4.0 with 16 lanes has a throughput of 256 Gbit/s (or 128 Gbit/s if two GPUs are behind a PCIe switch).

  • May combine two claims.

    Interpretationown data
    § Colin Raffel · p. 2

    Existing 100B+ models can run efficiently in PETALS with dynamic quantization, prioritizing low-latency connections, and load balancing between servers.

    “We demonstrate that existing 100B+ models can run efficiently in this setting with the help of several optimizations: dynamic quantization, prioritizing low-latency connections, and load balancing between servers (Section 3).”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Interpretationnone given
    § 4 Discussion and future work · p. 7

    Distributed parameter-efficient fine-tuning makes it easy for users to apply the base model to new tasks.

    “distributed parameterefficient fine-tuning makes it easy for users to apply the base model to new tasks.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation
    § Colin Raffel · p. 1

    BLOOM is a 176 billion parameter model supporting 46 natural and 13 programming languages.

    “Most recently, the BigScience project has released BLOOM, a 176 billion parameter model supporting 46 natural and 13 programming languages (Scao et al., 2022).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Scao et al., 2022

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Classified as probably not a substantial claim.

    Study design and conductnone given
    § 3.3 Benchmarks · p. 6

    The offloading numbers are based on the best possible hardware setup for offloading: CPU RAM offloading via PCIe 4.0 with 16 PCIe lanes per GPU and PCIe switches for pairs of GPUs.

    “We base our offloading numbers on the best possible hardware setup for offloading: CPU RAMoffloading via PCIe 4.0 with 16 PCIe lanes per GPU and PCIe switches for pairs of GPUs.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning
    § 3.1 Large model inference on consumer GPUs · p. 4

    For BLOOM, compressing weights to 8-bit precision reduces the number of required nodes from 44 to 22, which reduces latency in half and decreases the probability of a failure.

    “For BLOOM, this changes the number of required nodes from 44 to 22, which reduces latency in half and decreases the probability of a failure.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Interpretationnone given
    § 5 Conclusion · p. 7

    PETALS develops algorithms for reliable routing and load balancing.

    “In addition, we develop algorithms for reliable routing and load balancing.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Premisenone given
    § 2.2 Training for downstream tasks · p. 3

    Fine-tuning BLOOM-176B with Adam would require almost 3 TB of GPU memory to store model, gradients, and optimizer states.

    “For example, fine-tuning BLOOM-176B with Adam would require almost 3 TB of GPU memory to store model, gradients, and optimizer states.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisecitation
    § 2.2 Training for downstream tasks · p. 3

    Parameter-efficient fine-tuning approaches are often competitive with full model fine-tuning and even outperform it in low-data regimes.

    “Despite their lower memory requirements, parameter-efficient approaches are often competitive with full model fine-tuning (Hu et al., 2021; Liu et al., 2021a; Yong and Nikoulina, 2022) and even outperform it in low-data regimes (Liu et al., 2022b).”
    Quote in the paperNot a substantial claim0 of 3 sources checked
    • cites Hu et al., 2021

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Liu et al., 2021a

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Liu et al., 2022b

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

16 of 16

  • The paper relies on this without support.

    Generating one token at a time with BLOOM-176B takes at least 5.5 seconds for the fastest RAM offloading setup and 22 seconds for the fastest SSD offloading.

    “for example, generating one token at a time with BLOOM-176B takes at least 5.5 seconds for the fastest RAM offloading setup and 22 seconds for the fastest SSD offloading.”

  • The paper relies on this without support.

    APIs are often not flexible enough for research use: there is no way to change the model control flow or access internal states.

    “However, APIs are often not flexible enough for research use: there is no way to change the model control flow or access internal states.”

  • The authors state this as an assumption.

    Each server has at least 16 GB of CPU RAM and 8 GB of GPU memory.

    “We assume that each server has at least 16 GB of CPU RAM, 8 GB of GPU memory.”

  • The paper relies on this without support.

    OPT-175B and BLOOM-176B need over 350 GB accelerator memory for inference and significantly more for fine-tuning.

    “For instance, OPT-175B and BLOOM-176B need over 350 GB accelerator memory for inference and significantly more for fine-tuning.”

  • The authors state this as an assumption.

    Servers in the system are run by many independent parties.

    “We assume that servers in our system are run by many independent parties.”

  • The authors state this as an assumption.

    Offloading latency is zero for the upper bound estimation.

    “We assume an offloading latency of zero for the upper bound estimation.”

  • The paper relies on this without support.

    In 8-bit, the model uses 1 GB of memory per billion parameters.

    “In 8-bit, the model uses 1 GB of memory per billion parameters”

  • The paper relies on this without support.

    PCIe 4.0 with 16 lanes has a throughput of 256 Gbit/s (or 128 Gbit/s if two GPUs are behind a PCIe switch).

    “PCIe 4.0 with 16 lanes has a throughput of 256 Gbit/s (or 128 Gbit/s if two GPUs are behind a PCIe switch)”

  • The paper relies on this without support.

    Peers serving the first layers of the model can use their inputs to recover input tokens.

    “peers serving the first layers of the model can use their inputs to recover input tokens”
9 of 9

Cites

The publisher hasn’t deposited this paper’s reference list.

Research works printed but not linked: 5

  1. Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019. Parameter-efficient transfer learning for nlp. In International Conference on Machine Learning , pages 2790-2799. PMLR.
  2. Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. 2020. Retrieval-augmented generation for knowledgeintensive nlp tasks. In Advances in Neural Information Processing Systems , volume 33, pages 94599474. Curran Associates, Inc.
  3. Xiang Lisa Li and Percy Liang. 2021. Prefix-tuning: Optimizing continuous prompts for generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) , pages 4582-4597, Online. Association for Computational Linguistics.
  4. Max Ryabinin and Anton Gusev. 2020. Towards crowdsourced training of large neural networks using decentralized mixture-of-experts. In Advances in Neural Information Processing Systems , volume 33, pages 3659-3672. Curran Associates, Inc.
  5. Yi-Lin Sung, Varun Nair, and Colin Raffel. 2021. Training neural networks with fixed sparse masks. Advances in Neural Information Processing Systems .

Cited by

Sorted by citations

Related

Authors
Borzunov, A., Baranchuk, D., Dettmers, T., Ryabinin, M., Belkada, Y., Chumachenko, A., Samygin, P., & Raffel, C.
Affiliations
  • Yandex (Russia) (RU)
  • University of Washington (US)
  • Yandex School of Data Analysis (RU)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2022
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2209.01188
arXiv
2209.01188
OpenAlex
W4294782983
Cited by
6
References
0
Citations by year
2026: 1 · 2025: 2 · 2024: 2 · 2023: 1
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
Keywords
large language models, collaborative inference, fine-tuning, memory-efficient inference, hidden states, model extension

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-31.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-31.

Cited sources are checked by gpt-6-sol.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review