CoChat
Sign Up
PreprintCited by 10Parallel Computing and Optimization TechniquesStochastic Gradient Optimization TechniquesCloud Computing and Resource Management

SWARM Parallelism: Training Large Models Can Be Surprisingly Communication-Efficient

Open access
Ryabinin, Max, Dettmers, Tim, Diskin, Michael, Borzunov, Alexander·2023·arXiv
DOI 10.48550/arxiv.2301.11913arXiv 2301.11913
Open this paper in CoChat — free accountOpen PDF

Abstract

Many deep learning applications benefit from using large models with billions of parameters. Training these models is notoriously expensive due to the need for specialized HPC clusters. In this work, we consider alternative setups for training large models: using cheap "preemptible" instances or pooling existing resources from multiple regions. We analyze the performance of existing model-parallel algorithms in these conditions and find configurations where training larger models becomes less communication-intensive. Based on these findings, we propose SWARM parallelism, a model-parallel training algorithm designed for poorly connected, heterogeneous and unreliable devices. SWARM creates temporary randomized pipelines between nodes that are rebalanced in case of failure. We empirically validate our findings and compare SWARM parallelism with existing large-scale training approaches. Finally, we combine our insights with compression strategies to train a large Transformer language model with 1B shared parameters (approximately 13B before sharing) on preemptible T4 GPUs with less than 200Mb/s network.

AI snapshot

Written from the paper’s full text.

TL;DR

The paper analyzes how communication and computation scale with model size in pipeline parallelism, deriving a "Square-Cube Law" showing that larger models can be trained with less network overhead per unit of compute. It then proposes SWARM parallelism, a decentralized pipeline-parallel algorithm with stochastic wiring and adaptive rebalancing, and uses it to train a 1B-parameter (≈13B-equivalent) Transformer on preemptible T4 GPUs with under 200 Mb/s network.

Key findings

  • Larger models achieve better GPU utilization under the same network conditions: at 500 Mb/s, relative GPU utilization was 18.0% ("base"), 32.1% ("xxlarge"), 82.1% ("GPT-3"), and 89.5% ("Ours" with 8-bit compression); at 100 ms latency these were 2.78%, 14.9%, 60.2%, and 71.5% respectively (Table 1).
  • SWARM is competitive with HPC baselines even in idealized homogeneous conditions: for "GPT-3" (4 layers) throughput was 168.3 min/batch (no latency) vs 164.5 for GPipe and 272.7 for ZeRO-Offload; for the full "Ours" model, 432.2 vs 420.0 and 372.0 (Table 2).
  • Adaptive rebalancing substantially improves throughput: with T = 300 s, SWARM reached 95.8% of optimal throughput overall (88.9% in the last hour) vs 82.7% (45.4% in the last hour) with no rebalancing; T = 60 s reached 97.6% (Table 5).
  • With layer sharing and 8-bit compression, T4 GPUs reached 17.6 samples/s (best-case 19.2) with optimal bandwidth of 317.8 Mb/s upload / 397.9 Mb/s download; A100s reached 16.9 (best-case 25.5) (Table 3). Training convergence on the Pile over ~4 weeks with 400 preemptible T4s matched a DDP baseline on 128 A100s (Figure 4).

Methods

  • Design: SWARM parallelism partitions the model into evenly sized pipeline stages ("swarms"); peers within a stage serve the same layers, and stochastic wiring routes each microbatch to peers in proportion to measured throughput using Interleaved Weighted Round-Robin scheduling, with peers discovered via a Distributed Hash Table. Adaptive rebalancing moves peers between stages every T seconds based on queue-size load measurements, and failed peers are banned and rerouted.
  • Data/experiments: controlled V100 experiments measuring GPU utilization and network usage across four Transformer configurations (base, xxlarge, GPT-3-scale, and a modified "Ours" with 8-bit quantized activations); a comparison against GPipe and ZeRO-Offload over 16 workers with 4 Transformer layers; large-scale training of a 1.01B-parameter (≈13B-equivalent via layer sharing) Transformer on the Pile with 400 preemptible T4 instances and 7×8 A100 instances; and a simulation of rebalancing using 32 hours of real T4 node-count statistics.
  • Analysis: throughput (time to process 6250 sequences of 512 tokens) and All-Reduce time; relative throughput vs a theoretically optimal rebalancing strategy; scaling of throughput with node count (8–128 nodes) and with number of stages (4–32).

Limitations

  • Stated: SWARM is by default worse than traditional parallelism in supercomputers due to extra complexity, and is only useful there for heterogeneous devices; for models under 1B parameters, sharded data-parallel approaches can be more optimal; SWARM suffers from the pipeline "bubble" problem with long pipelines; asynchronous updates were not investigated due to potential convergence issues; rebalancing moves only one peer at a time; and throughput drops as stage count increases because variance in preemption is too large relative to peer counts.
  • Evident from the text: the rebalancing evaluation (Section 4.4) is a simulation using recorded node-count statistics rather than a live deployment; the large-scale convergence comparison is run for approximately 4 weeks and compared against a single DDP baseline; and the paper notes that its GPU-utilization measurements are not directly comparable to peak-FLOP/s utilization reported by other systems (Appendix A).

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

DataCite · 10.48550/arxiv.2301.11913

Cites

0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

0 of 12 cited sources checked · 0 supported · 0 flagged

  • Findingown data

    In the earlier Figure 1 experiment, the same model spent most of the time waiting for communication between pipeline stages.

    “where the same model spent most of the time waiting for the communication between pipeline stages.”
    Quote in the paperNot a substantial claim
  • Findingown data
    § I. Compression-Aware Architectures

    Blockwise (de)quantization takes less than 1% of total computation time.

    “However, this is not required, as blockwise (de)quantization takes less than 1% of total computation time: see Appendix J for details.”
    Quote in the paperNot a substantial claim
  • Findingown data
    § C. Stochastic Wiring Details

    The same bandwidth allows for >80% GPU utilization when dealing with GPT-3-sized layers.

    “However, that same bandwidth allows for > 80% GPU utilization when dealing with GPT-3-sized layers.”
    Quote in the paperSubstantial claim
  • Findingown data
    § C. Stochastic Wiring Details

    If 3 GPT-3 layers are colocated per pipeline stage, GPU utilization can be further improved to >90%.

    “If we colocate 3”
    Quote in the paperNot a substantial claim
  • Findingown data
    § 0 10B 20B 30B 40B 50B

    The training dynamics of SWARM parallelism and regular data-parallel training with offloading are similar over approximately 4 weeks of training.

    “it can be seen that the training dynamics of two approaches are indeed similar”
    Quote in the paperSubstantial claim
  • May combine two claims. 19
  • The quote only partly states this claim. 3
  • Classified as probably not a substantial claim. 27

  • May combine two claims.

    Findingown data
    § I. Compression-Aware Architectures

    8-bit compression does not degrade the time to 22 perplexity and maintains close to the final perplexity of the baseline.

    “We can see that 8-bit compression does not degrade the time to 22 perplexity and maintains close to the final perplexity of the baseline.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § I. Compression-Aware Architectures

    Adding a bottleneck only adds two linear layers for the forward pass and decreases the size of MLP activations; its computational overhead is negligible (less than 1% for typical sizes).

    “Note that adding a bottleneck only adds two linear layers for the forward pass and decreases the size of MLP activations; thus, its computational overhead is negligible (less than 1% for typical sizes, see Appendix J).”
    Quote in the paperSubstantial claim

  • The quote only partly states this claim.

    Findingreasoning
    § C. Stochastic Wiring Details

    Training performance exhibits an approximately linear scaling pattern with the number of nodes from 8 to 128.

    “It can be seen that the training performance exhibits an approximately linear scaling pattern”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given
    § Supplementary Material

    Some configurations in Section 4.1 measure less than 20% GPU idle time.

    “Some configurations in Section 4.1 measure less than 20% GPU idle time”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data
    § 4 stages

    For approximately 4.5% of the experiment, the performance of the unbalanced pipeline exceeded the throughput of the balanced one due to random choice of disconnecting peers.

    “However, this held true only for ≈ 4.5% of the experiment and was quickly mitigated by adaptive rebalancing.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § 4. Experiments

    Larger models achieve better GPU utilization rate in the same network conditions, since their communication load grows slower than computation.

    “As depicted in Figure 1 (right) and Figure 3, larger models achieve better GPU utilization rate in the same network conditions, since their communication load grows slower than computation.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § 4 stages

    Rebalancing partially mitigates the performance drop with more stages: the rebalanced pipeline regains its performance over time and achieves a higher overall throughput than the baseline.

    “Still, rebalancing allows to partially mitigate the issue: while we observe a more consistent downward trend for the baseline strategy, the rebalanced pipeline regains its performance over time and achieves a higher overall throughput.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § I. Compression-Aware Architectures

    While the bottleneck architecture is competitive with maxout for a compression factor of 2x with two stages, maxout has better perplexities if more stages or a higher compression ratio is used.

    “We can see that while the bottleneck architecture is competitive with maxout for a compression factor of 2x with two stages, maxout has better perplexities if more stages or a higher compression ratio is used.”
    Quote in the paperSubstantial claim

  • May combine two claims. The quote only partly states this claim.

    Findingown data
    § 4 stages

    Both the rebalancing and the no-rebalancing strategies drop in performance as the number of pipeline stages increases from 4 to 32.

    “both strategies drop in performance with the increase in the stage count”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § I. Compression-Aware Architectures

    The compression-aware bottleneck and maxout architectures perform equal to each other, but degrade final perplexity slightly and increase time to a perplexity of 22 by 26-28%.

    “The compression-aware bottleneck and maxout architectures perform equal to each other, but degrade final perplexity slightly and increase time to a perplexity of 22 by 26–28%.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § 4 stages

    Decreasing T from 300 to 60 seconds improves both the overall throughput and the speed of convergence to optimal pipeline performance.

    “As expected, decreasing T from 300 to 60 seconds improves both the overall throughput and the speed of convergence to optimal pipeline performance.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Interpretationnone given

    If any single pipeline stage loses too many peers, the remaining ones will face an increased processing load, which will inevitably form a bottleneck.

    “If any single pipeline stage loses too many peers, the remaining ones will face an increased processing load, which will inevitably form a bottleneck.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: Workers can join and leave training at any time.

  • Probably not a substantial claim: may not pass S4 (based).

    Premisenone given
    § Supplementary Material

    All-to-all communication latency is determined by the slowest peer.

    “This is exacerbated further, because all-to-all communication latency is determined by the slowest peer.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Interpretationnone given
    § Abstract

    SWARM parallelism is a model-parallel training algorithm designed for poorly connected, heterogeneous and unreliable devices.

    “we propose SWARM parallelism1, a model-parallel training algorithm designed for poorly connected, heterogeneous and unreliable devices.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The training devices are poorly connected, heterogeneous, and unreliable.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning
    § Supplementary Material

    Using long pipelines can result in a reduced training throughput.

    “While its design allows for any number of stages, using long pipelines can result in a reduced training throughput.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductown data

    The adaptive rebalancing evaluation ran 10 simulations with different random seeds and averaged the resulting trajectories.

    “We run 10 simulations with different random seeds and average the resulting trajectories.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Interpretationnone given
    § Supplementary Material

    The data loader state can be recomputed from the last known SGD step.

    “Similarly, the data loader state can be recomputed from the last known SGD step.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductnone given
    § C. Stochastic Wiring Details

    All results in Section 4.1 are averaged over 1000 consecutive batches, and the standard deviations are below 0.1%.

    “All results are averaged over 1000 consecutive batches; the standard deviations are below 0.1%.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationown data
    § Supplementary Material

    Training with SWARM can accelerate training while also allowing to train larger models.

    “Training with SWARM can accelerate training while also allowing to train larger models; see Appendix E for a detailed comparison.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning

    ZeRO-Offload takes longer to aggregate gradients because each peer must aggregate the entire model, whereas in SWARM and GPipe peers aggregate a single pipeline stage.

    “likely because each peer must aggregate the entire model, whereas in SWARM and GPipe, peers aggregate a single pipeline stage.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Interpretationreasoning
    § C. Stochastic Wiring Details

    All surviving peers use non-stale aggregated gradients over the specified batch size when performing the optimizer step.

    “This procedure ensures that all surviving peers use non-stale aggregated gradients over the specified batch size when performing the optimizer step.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Training large models over heterogeneous devices can be optimized with global scheduling.

    “Moreover, training large models over heterogeneous devices can be optimized with global scheduling (Yuan et al., 2022).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Yuan et al., 2022

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning
    § C. Stochastic Wiring Details

    The total complexity of Algorithm 2 is O(MS), where M is the highest number of peers over all stages and S is the number of stages.

    “These operations are nested in the loop over all stages, which means that the total complexity of the algorithm is O(MS).”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Premisenone given
    § Supplementary Material

    Bottleneck and maxout compression are general compression techniques that can be applied to any layer in any architecture.

    “Bottleneck and maxout compression are general compression techniques that can be applied to any layer in any architecture.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Interpretationnone given

    If any node fails or leaves training prematurely, it will stall the entire training procedure.

    “On top of that, if any node fails or leaves training prematurely, it will stall the entire training procedure.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Workers can join and leave training at any time.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Interpretationreasoning

    Training with small batch sizes is more sensitive to the All-Reduce time, since the algorithm needs to run All-Reduce more frequently, and vice versa.

    “Intuitively, training with small batch sizes is more sensitive to the All-Reduce time (since the algorithm needs to run All-Reduce more frequently) and vice versa.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Layer sharing reduces the number of aggregated gradients by a factor of how many times each layer is reused.

    “Besides gradient compression, another effective technique is to use layer sharing (Lan et al., 2020), which reduces the number of aggregated gradients by a factor of how many times each layer is reused.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Lan et al., 2020

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Classified as probably not a substantial claim.

    Premisenone given

    The compute time for naïve matrix multiplication scales as O(n^3).

    “The compute time for naïve matrix multiplication scales as O(n3).”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). The quote only partly states this claim.

    Interpretationnone given
    § Supplementary Material

    For smaller models, a sharded data-parallel approach can be more optimal than SWARM.

    “For smaller models, a sharded data-parallel approach can be more optimal.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Premisecitation
    § Supplementary Material

    Narayanan et al. (2021) measures GPU utilization as a fraction of theoretical peak FLOP/s of their GPUs.

    “Narayanan et al. (2021) measures GPU utilization as a fraction of theoretical peak FLOP/s of their GPUs.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Narayanan et al. (2021)

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Interpretationreasoning
    § Supplementary Material

    As a rule of thumb, SWARM will run at a 20% slower speed than systems described by Narayanan et al. (2021) using infrastructure that is several times cheaper.

    “As a rule of thumb, one can say that SWARM will run at a 20% slower speed than systems described by Narayanan et al. (2021) using the infrastructure that is several times cheaper.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Narayanan et al. (2021)

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Interpretationreasoning
    § Supplementary Material

    As long as there is at least one operational peer at every pipeline stage and at least one trainer, SWARM can work without any issues.

    “As long as there is at least one operational peer at every pipeline stage and at least one trainer, SWARM can work without any issues.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Interpretationnone given
    § Supplementary Material

    Elastic sharded data parallelism is better than SWARM if the number of expected failures is relatively low.

    “Elastic sharded data parallelism is better than SWARM if the number of expected failures is relatively low.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Dettmers (2016) uses 8-bit quantization to compress gradients before communication.

    “Alternatively, Dettmers (2016) uses 8-bit quantization to compress gradients before communication.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Dettmers, 2016

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Interpretationreasoning
    § I. Compression-Aware Architectures

    If network bandwidth is unlimited, maxout or bottleneck compression would result in slower training.

    “However, the same two techniques would result in slower training in a setup where network bandwidth is unlimited.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisenone given
    § Supplementary Material

    SWARM's design allows for any number of pipeline stages.

    “While its design allows for any number of stages, using long pipelines can result in a reduced training throughput.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning
    § Supplementary Material

    For low-bandwidth devices with 100 Mb/s speed, sharded data parallelism would entail an overhead of 5.5 minutes per forward/backward pass, which is difficult to overlap with computation.

    “For low-bandwidth devices with 100 Mb/s speed, this would entail an overhead of 5.5 minutes per forward/backward pass, which is difficult to overlap with computation.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Premisecitation
    § I. Compression-Aware Architectures

    Dynamic 8-bit quantization with blockwise scaling reduces communication by approximately 2x and 4x for half and full precision, respectively.

    “Instead, we use dynamic 8-bit quantization with blockwise scaling from (Dettmers et al., 2022). This technique reduces communication by ≈2x and ≈4x for half and full precision, respectively.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Dettmers et al., 2022

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Deep Gradient Compression sparsifies gradients and corrects momentum after synchronization.

    “Deep Gradient Compression (Lin et al., 2018) sparsifies the gradients and corrects the momentum after synchronization”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Lin et al., 2018

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Interpretationnone given
    § Supplementary Material

    The effectiveness of bottleneck and maxout compression may vary depending on where in the model they are applied and what kind of model they are applied to.

    “However, their effectiveness may vary depending on where in the model they are applied and what kind of model these are applied to (for example, CNNs vs. RNNs vs. Transformers).”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning
    § Supplementary Material

    SWARM can be useful in case of supercomputers that have heterogeneous devices.

    “However, SWARM can be useful in case of supercomputers that have heterogeneous devices.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Interpretationnone given
    § Supplementary Material

    Without layer sharing, the overhead within each pipeline stage to synchronize the gradients is increased by the number of layers not being shared.

    “Without layer sharing, the overhead within each pipeline stage to synchronize the gradients is increased by the number of layers not being shared.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning
    § C. Stochastic Wiring Details

    SWARM produces exactly the same stepwise updates as conventional distributed training algorithms and will therefore achieve a solution in the same number of steps.

    “That is, SWARM produces exactly the same stepwise updates as conventional distributed training algorithms and will therefore achieve a solution in the same number of steps.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § Supplementary Material

    Many HPC systems only achieve approximately 80% GPU utilization.

    “while many HPC systems only achieve ≈ 80% GPU utilization.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Narayanan et al. (2021)

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

44 of 44

  • The paper relies on this without support.

    Workers can join and leave training at any time.

    “As we described in Section 2.2, our workers can join and leave training at any time.”

  • The paper relies on this without support.

    The training devices are poorly connected, heterogeneous, and unreliable.

    “we propose SWARM parallelism1, a model-parallel training algorithm designed for poorly connected, heterogeneous and unreliable devices.”

  • The authors state this as an assumption.

    In the rebalancing simulation, when a worker is removed from the pipeline, the stage it was removed from is chosen uniformly at random over the four pipeline stages.

    “When a worker is removed from the pipeline, we randomly choose the stage it was removed from: that is, removing N peers corresponds to N samples from the uniform distribution over four pipeline stages.”

  • The paper relies on this without support.

    Executing a single pipeline stage 3 times locally on a single server and multiplying the single-node estimates by the number of nodes yields a valid estimate of the ideal best-case pipeline performance with infinite bandwidth and zero latency.

    “The ideal case is emulated by executing a single pipeline stage 3 times locally on a single server and multiplying the single-node estimates by the number of nodes.”

  • The authors state this as an assumption.

    The i.i.d. assumption in the large-scale training setup holds, so the distribution of gradients is expected to be the same across examples within a batch.

    “because of the i.i.d. assumption in the large-scale training setup, the distribution of gradients is expected to be the same.”

  • The paper relies on this without support.

    Any realistic deep learning workload cannot achieve 100% peak FLOP/s.

    “Since any realistic deep learning workload cannot achieve 100% peak FLOP/s”

  • The paper relies on this without support.

    The square-cube law for distributed training holds.

    “When considered together with the square-cube law for distributed training and SWARM parallelism”
7 of 7

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Dynamic blockwise quantization halves the bandwidth requirements without any noticeable effect on generation quality.

    “Dynamic blockwise quantization halves the bandwidth requirements without any noticeable effect on generation quality.”

    From Petals: Collaborative Inference and Fine-tuning of Large Models (2022) · cites this paper as Ryabinin et al., 2023

    In Appendix I (p. 23), Table 7 reports that dynamic blockwise 8-bit quantization halved data transfer relative to the 16-bit baseline and produced similar final perplexity (21.13 versus 21.02). The paper evaluates perplexity, not generated-output quality, and the halving applies to activation data transfer rather than necessarily to total bandwidth requirements. (LLM, conf=0.85)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • Figures label the "Ours" pipeline stage as 12 layers, but the setup says 3

    Minor

    Figures 1 and 3 label the "Ours" configuration as having 12 layers at a stage, while the experiment description says each of its four servers hosts 3 layers. The discrepancy affects the setup behind the reported GPU-utilization result.

  • Appendix describes four GPUs for configurations using twelve

    Minor

    Appendix F says all four GPUs were on different servers, but its setup specifies 12 single-GPU servers for each of the first three configurations. Four GPUs describes only the "Ours" configuration, not all four benchmarked configurations.

2 of 2

Cites

The publisher hasn’t deposited this paper’s reference list.

Part of this paper’s reference list could not be split into single entries.

Research works printed but not linked: 1

  1. [148]148. IEEE, 2020. Zheng, L., Li, Z., Zhang, H., Zhuang, Y., Chen, Z., Huang, Y., Wang, Y., Xu, Y., Zhuo, D., Xing, E. P., Gonzalez, J. E., and Stoica, I. Alpa: Automating inter- and intra- operator parallelism for distributed deep learning, 2022. URL https://arxiv.org/abs/2201.12023. 17 https://openreview.net/forum?id=UHoGOaGjEq https://openreview.net/forum?id=UHoGOaGjEq https://arxiv.org/abs/2106.04560 https://arxiv.org/abs/2106.04560 https://doi.org/10.1007/978-3-319-20119-1_2 https://doi.org/10.1007/978-3-319-20119-1_2 https://arxiv.org/abs/2201.12023 SWARM Parallelism: Training Large Models Can Be Surprisingly Communication-Efficient · DOI

Cited by

Sorted by citations

Related

Authors
Ryabinin, M., Dettmers, T., Diskin, M., & Borzunov, A.
Affiliations
  • National Research University Higher School of Economics (RU)
  • University of Washington (US)
Venue
arXiv
Publisher
arXiv
Published
2023
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2301.11913
arXiv
2301.11913
OpenAlex
W4318621310
Cited by
8
References
0
Citations by year
2025: 3 · 2024: 3 · 2023: 2
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Hardware and Architecture › Parallel Computing and Optimization Techniques
  • Physical Sciences › Computer Science › Artificial Intelligence › Stochastic Gradient Optimization Techniques
  • Physical Sciences › Computer Science › Information Systems › Cloud Computing and Resource Management
Keywords
communication efficiency, heterogeneous devices, pipeline parallelism, parameter sharing, transformer language models, network communication, fault tolerance

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-31.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-31.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review