CoChat
Sign Up
Cited by 1,725Privacy-Preserving Technologies in DataStochastic Gradient Optimization TechniquesMachine Learning and Data Classification

Robust and Communication-Efficient Federated Learning From Non-i.i.d. Data

Open access
Sattler, Felix, Wiedemann, Simon, Muller, Klaus-Robert, Samek, Wojciech·2020·IEEE Transactions on Neural Networks and Learning Systems
DOI 10.1109/tnnls.2019.2944481
Open this paper in CoChat — free accountOpen PDF

Abstract

Federated learning allows multiple parties to jointly train a deep learning model on their combined data, without any of the participants having to reveal their local data to a centralized server. This form of privacy-preserving collaborative learning, however, comes at the cost of a significant communication overhead during training. To address this problem, several compression methods have been proposed in the distributed training literature that can reduce the amount of required communication by up to three orders of magnitude. These existing methods, however, are only of limited utility in the federated learning setting, as they either only compress the upstream communication from the clients to the server (leaving the downstream communication uncompressed) or only perform well under idealized conditions, such as i.i.d. distribution of the client data, which typically cannot be found in federated learning. In this article, we propose sparse ternary compression (STC), a new compression framework that is specifically designed to meet the requirements of the federated learning environment. STC extends the existing compression technique of top- k gradient sparsification with a novel mechanism to enable downstream compression as well as ternarization and optimal Golomb encoding of the weight updates. Our experiments on four different learning tasks demonstrate that STC distinctively outperforms federated averaging in common federated learning scenarios. These results advocate for a paradigm shift in federated optimization toward high-frequency low-bitwidth communication, in particular in the bandwidth-constrained learning environments.

AI snapshot

Written from the paper’s full text.

TL;DR

The paper proposes sparse ternary compression (STC), a federated learning protocol that compresses both upstream and downstream updates via sparsification, ternarization, error accumulation, and Golomb encoding. Across four benchmarks, STC is more robust than federated averaging and signSGD to non-i.i.d. client data and requires fewer communicated bits to reach target accuracy.

Key findings

  • STC compresses both upload and download; at sparsity \(p=1/400\) it compresses updates by roughly \(\times1050\), and ternarization adds \(H_{\text{sparse}}/H_{\text{STC}}=4.414\) at \(p=0.01\).
  • In the extreme non-i.i.d. case where every client holds data from exactly one class, STC achieves 79.5% and 53.2% accuracy at full and partial client participation, respectively, while both federated averaging and signSGD fail to converge.
  • At batch size 1, STC reaches 63.8% accuracy after 20,000 iterations, versus 39.2% for federated averaging.
  • On CIFAR in an i.i.d. environment, STC at \(p=0.0025\) communicates 183.9 MB to target accuracy, a \(\times199.5\) reduction versus the 36,696 MB baseline and less than federated averaging (\(n=100\)) at 1,606 MB; downstream sparsification reduces final accuracy by at most 3% compared with upload-only sparsity.

Methods

  • The paper introduces STC with client-side and server-side residual error accumulation, server-side compression for downstream communication, a caching mechanism for partial client participation, and Golomb encoding of sparse ternary updates; it provides a convergence analysis via a \(k\)-contraction argument.
  • Experiments cover four tasks: VGG11* on CIFAR, a CNN on keyword spotting, an LSTM on Fashion-MNIST, and logistic regression on MNIST; the base federated setup uses 100 clients, 10% participation, local batch size 20, and 10 classes per client.
  • STC at \(p=1/400\) is compared with federated averaging at delay \(n=400\) and signSGD with coordinatewise step size \(\delta=0.0002\), evaluating accuracy and communicated bits across non-i.i.d.-ness, batch size, participation, and unbalancedness.

Limitations

  • Stated: partial client participation causes client residuals to go out of sync and increases gradient staleness; the download size grows linearly with the number of skipped rounds, with average skipped rounds equal to the inverse participation fraction, though the downlink is usually cheaper.
  • Stated: STC is particularly suited to low-latency, low-bandwidth channels; federated averaging may be preferred when communication is latency-constrained or participation is very low, and momentum should be avoided with small batches or non-i.i.d. data under low participation.
  • Evident from the text: evaluation is limited to four specified benchmarks, architectures, and hyperparameter settings; privacy leakage from model updates is acknowledged but not addressed or evaluated, with homomorphic encryption and differential privacy only cited as possible additional mechanisms.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1109/tnnls.2019.2944481

Cites

39 of 39 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

0 of 17 cited sources checked · 0 supported · 0 flagged

  • Findingnone given
    § C. Weight Update Caching for Partial Client Participation

    The average number of skipped rounds is equal to the inverse participation fraction 1/η.

    “The average number of skipped rounds is equal to the inverse participation fraction 1/η.”
    Quote in the paperNot a substantial claim
  • Findingnone given
    § III. CONTRIBUTION

    The results on STC also extend to the i.i.d. regime.

    “These results also extend to the i.i.d.regime.”
    Quote in the paperNot a substantial claim
  • Findingnone given
    § C. Weight Update Caching for Partial Client Participation

    For signSGD, the size of the downstream update only grows logarithmically with the delay period according to H(P(τ)_signSGD) ≤ log2(2τ+1).

    “This is also the case for signSGD although here the size of the downstream update only grows logarithmically with the delay period according to H (P (τ ) signSG D ) ≤ log 2 (2τ + 1).”
    Quote in the paperSubstantial claim
  • Findingreasoning
    § 3) Top-k Sparsification:

    For VGG11 on CIFAR, top-k sparsification training still converges reliably even if every client only holds data from exactly one class.

    “For VGG11 on CIFAR the training still converges reliably even if every client only holds data from exactly one class”
    Quote in the paperSubstantial claim
  • Findingreasoning
    § III. CONTRIBUTION

    None of the existing methods proposed for communication-efficient federated learning satisfies all requirements R1-R3.

    “In this article, we will demonstrate that none of the existing methods proposed for communication-efficient federated learning satisfies all of these requirements (see Table I).”
    • Also stated in: § 3) Top-k Sparsification: · plain
      “None of the existing compression methods supports both download compression and properly works with non-i.i.d.data.”
    Quote in the paperSubstantial claim
  • May combine two claims. 26
  • Classified as probably not a substantial claim. 14
  • The quote only partly states this claim. 3

  • May combine two claims.

    Findingown data
    § 1) Federated Averaging:

    For i.i.d. batches, the gradient sign becomes increasingly accurate with growing batch sizes; for non-i.i.d. batches, the gradient signs remain highly incongruent with the full-batch gradient no matter the batch size.

    “For i.i.d.batches, the gradient sign becomes increasingly accurate with growing batch sizes.For non-i.i.d.batches of data, this is not the case.The gradient signs remain highly incongruent with the full-batch gradient, no matter how large the size of the batch.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § B. Results

    In preliminary experiments, all compression methods achieve comparably fast convergence in terms of gradient evaluations on i.i.d. data, closely matching the uncompressed baseline, but suffer considerably in non-i.i.d. training settings.

    “We observe that while all compression methods achieve comparably fast convergence in terms of gradient evaluations on i.i.d.data, closely matching the uncompressed baseline (black line), they suffer considerably in the non-i.i.d.training settings.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § Algorithm 1 STC

    Additional ternarization has only a negligible effect on convergence speed and sometimes even increases the final accuracy of the trained model.

    “As we can see, additional ternarization does only have a negligible effect on the convergence speed and sometimes does even increase the final accuracy of the trained model.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § 1) Federated Averaging:

    At batch size 1, g1w is a very bad predictor of the true gradient sign with high variance and average congruence α(1)=0.51, just slightly higher than random.

    “As we can see, at a batch size of 1, g 1 w is a very bad predictor of the true gradient sign with a very high variance and an average congruence of α(1) = 0.51 just slightly higher than random.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § 1) Federated Averaging:

    SignSGD completely fails to converge on the CIFAR benchmark in the non-i.i.d. learning environment, and even for the convex logistic regression objective the training plateaus at a substantially degraded accuracy.

    “The method completely fails to converge on the CIFAR benchmark, and even for the convex logistic regression objective, the training plateaus at a substantially degraded accuracy.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data
    § Models and Data Sets:

    The VGG11* network achieves 85.46% accuracy on the validation set after 20,000 iterations of training with a constant learning rate of 0.16 and contains 865,482 parameters.

    “The resulting VGG11* network still achieves 85.46% accuracy on the validation set after 20 000 iterations of training with a constant learning rate of 0.16 and contains 865 482 parameters.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § B. Extending to Downstream Compression

    Sparsifying the download is not very harmful to convergence and decreases accuracy by at most 2% in both the i.i.d. and non-i.i.d. case, as long as download and upload sparsity are of the same order.

    “As we can see, for as long as download and upload sparsity are of the same order, sparsifying the download is not very harmful to the convergence and decreases the accuracy by at most 2% in both the i.i.d. and the non-i.i.d.case.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingreasoning
    § D. Lossless Encoding

    Golomb encoding reduces the average number of position bits to bpos; for p=0.01, bpos=8.38, translating to ×1.9 compression compared to a naive distance encoding with 16 fixed bits.

    “For a sparsity rate of e.g., p = 0.01, we get bpos = 8.38, which translates to ×1.9 compression, compared to a naive distance encoding with 16 fixed bits.”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: The sparsity pattern of the weight updates is random.

  • May combine two claims.

    Findingreasoning
    § Algorithm 1 STC

    Ternarization reduces the entropy of the update from H_sparse to H_STC; at sparsity rate p=0.01, the additional compression achieved by ternarization is H_sparse/H_STC=4.414.

    “At a sparsity rate of p = 0.01, the additional compression achieved by ternarization is H sparse /H STC = 4.414.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § B. Non-i.i.d.-ness of the Data

    The convergence behavior of both federated averaging and signSGD is very sensitive to the degree of i.i.d.ness of local client data, whereas sparse communication seems to be more robust.

    “Our preliminary experiments in Section V have already demonstrated that the convergence behavior of both federated averaging and signSGD is very sensitive to the degree of i.i.d.ness of the local client data, whereas sparse communication seems to be more robust.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: i.i.d. distribution of client data typically cannot be found in federated learning.

  • May combine two claims.

    Findingown data
    § 1) Local Batch Size:

    At an extreme batch size of one, the model trained with STC still achieves an accuracy of 63.8%, while the federated averaging model only reaches 39.2% after 20 000 training iterations.

    “At an extreme batch size of one, the model trained with STC still achieves an accuracy of 63.8%, while the federated averaging model only reaches 39.2% after 20 000 training iterations.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § VIII. LESSONS LEARNED

    Momentum optimization harms performance when clients train with small batch sizes or the client data are non-i.i.d. and the participation rate is low.

    “Momentum optimization should be avoided in federated learning whenever either clients are training with small batch sizes or the client data are non-i.i.d. and the participation rate is low (see Figs. 678).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § D. Communication Efficiency

    On the CIFAR learning task, STC at a sparsity rate of p = 0.0025 only communicates 183.9 MB worth of data, which is a reduction in communication by a factor of ×199.5 as compared to the baseline which requires 36696 MB and federated averaging (n = 100), which still requires 1606 MB.

    “On the CIFAR learning task, STC at a sparsity rate of p = 0.0025 only communicates 183.9 MB worth of data, which is a reduction in communication by a factor of ×199.5 as compared to the baseline with requires 36696 MB and federated averaging (n = 100), which still requires 1606 MB.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given
    § Compression Methods:

    A delay period of n = 400 iterations for federated averaging results in a compression rate of ×400.

    “A delay period of n = 400 iterations for federated averaging results in a slightly smaller compression rate of ×400.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § 1) Local Batch Size:

    If only 5 out of 400 clients participate in every round, STC (without momentum) still manages to achieve an accuracy of 68.2% while federated averaging stagnates at 42.3% accuracy.

    “If only 5 out of 400 clients participate in every round, STC (without momentum) still manages to achieve an accuracy of 68.2% while federated averaging stagnates at 42.3% accuracy.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § 1) Local Batch Size:

    Reducing the participation rate has negative effects on both federated averaging and STC.

    “As we can see, reducing the participation rate has negative effects on both federated averaging and STC.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § 1) Local Batch Size:

    Even if the data are highly concentrated on a few clients (γ = 0.9), all methods converge reliably, and for federated averaging, the accuracy even slightly goes down with increased balancedness.

    “Even if the data are highly concentrated on a few clients (as is the case for γ = 0.9), all methods converge reliably, and for federated averaging, the accuracy even slightly goes down with increased balancedness.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § B. Non-i.i.d.-ness of the Data

    In the extreme case where every client only holds data from exactly one class, STC still achieves 79.5% and 53.2% accuracy at full and partial client participations, respectively, while both federated averaging and signSGD fail to converge at all.

    “In the extreme case where every client only holds data from exactly one class, STC still achieves 79.5% and 53.2% accuracy at full and partial client participations, respectively, while both federated averaging and signSGD fail to converge at all.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: i.i.d. distribution of client data typically cannot be found in federated learning.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data
    § Algorithm 1 STC

    Preliminary experiments are in line with the theoretical findings on STC convergence.

    “Preliminary experiments are in line with our theoretical findings.”
    Quote in the paperNot a substantial claim

  • The quote only partly states this claim.

    Interpretationown data
    § III. CONTRIBUTION

    The sparse ternary compression (STC) protocol meets all requirements R1-R3.

    “We will then proceed to construct a new efficient communication protocol for federated learning that resolves these issues and meets all requirements (R1)-(R3).”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § II. CHALLENGES OF THE FEDERATED LEARNING ENVIRONMENT

    Federated learning environments may constitute multiple millions of participants.

    “Federated learning environments may constitute of multiple millions of participants [18].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [18]

      Not checked in the free run.

  • Classified as probably not a substantial claim.

    Premisecitation
    § I. INTRODUCTION

    Federated learning allows multiple parties to jointly train a deep learning model on their combined data without revealing their data to a centralized server.

    “Federated learning resolves this issue as it allows multiple parties to jointly train a deep learning model on their combined data, without any of the participants having to reveal their data to a centralized server [10].”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [10]

      Not checked in the free run.

  • May combine two claims.

    Interpretationreasoning
    § V. LIMITATIONS OF EXISTING COMPRESSION METHODS

    A non-i.i.d. distribution of local data violates assumption (2), renders existing convergence guarantees inapplicable, and has dramatic effects on the practical performance of communication-efficient distributed training algorithms.

    “As it violates assumption (2), a non-i.i.d.distribution of the local data renders existing convergence guarantees, as formulated in [19]- [21] and [29], inapplicability and has dramatic effects on the practical performance of communicationefficient distributed training algorithms as we will demonstrate in the following experiments.”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites [19]-[21]

      Not checked in the free run.

    • cites [29]

      Not checked in the free run.

  • May combine two claims.

    Premisecitation
    § I. INTRODUCTION

    Naively following the federated learning protocol, every participating client must communicate a full model update during every training iteration, and each update is the same size as the trained model.

    “When naively following the protocol described earlier, every participating client has to communicate a full model update during every training iteration.Every such update is of the same size as the trained model, which can be in the range of gigabytes for modern architectures with millions of parameters [15], [16].”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites [15] source

      Not checked in the free run.

    • cites [16] source

      Not checked in the free run.

  • May combine two claims.

    Interpretationreasoning
    § Algorithm 1 STC

    For T ∈ O((d/k)((L/μ))^{1/2}), STC converges at rate O((G^2/μT)), which is the same as for regular SGD.

    “This means that for T ∈ O((d/ k)((L/μ)) 1/2 ), STC converges at rate O((G 2 /μT )), which is the same as for regular SGD!”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: The objective function is L-smooth, μ-strongly convex, and has bounded gradients.

  • The quote only partly states this claim.

    Interpretationreasoning
    § C. Weight Update Caching for Partial Client Participation

    All compression methods that communicate only parameter updates instead of full models suffer from the same problem of download size growing with skipped rounds.

    “Essentially, all compression methods that communicate only parameter updates instead of full models suffer from this same problem.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Premisenone given
    § V. LIMITATIONS OF EXISTING COMPRESSION METHODS

    Related work on efficient distributed deep learning almost exclusively considers i.i.d. data distributions among clients, assuming unbiasedness of local gradients with respect to the full-batch gradient.

    “The related work on efficient distributed deep learning almost exclusively considers i.i.d.data distributions among the clients, i.e., they assume unbiasedness of the local gradients with respect to the full-batch gradient according to”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisereasoning
    § B. Extending to Downstream Compression

    Existing compression frameworks for distributed training only compress client-to-server communication, which is sufficient for all-reduce but not feasible in federated learning where clients download aggregated updates, leading to a communication bottleneck.

    “Existing compression frameworks that were proposed for distributed training (see [19], [20], [23], [25]) only compress the communication from clients to the server, which is sufficient for applications where aggregation can be achieved via an all-reduce operation.However, in the federated learning setting, where the clients have to download the aggregated weight-updates from the server, this appro”
    Quote in the paperSubstantial claim0 of 3 sources checked
    • cites [19]

      Not checked in the free run.

    • cites [20]

      Not checked in the free run.

    • cites [23]

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S4 (based). May combine two claims.

    Premisenone given
    § II. CHALLENGES OF THE FEDERATED LEARNING ENVIRONMENT

    In federated learning, the distribution of both training data and computational resources is a fundamental and fixed property of the learning environment.

    “In federated learning, the distribution of both training data and computational resources is a fundamental and fixed property of the learning environment.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationcitation
    § 1) Local Batch Size:

    If a nonrepresentative subset of clients is selected to participate in a particular communication round of federated averaging, this can steer the optimization process away from the minimum and might even cause catastrophic forgetting of previously learned concepts.

    “If a nonrepresentative subset of clients is selected to participate in a particular communication round of federated averaging, this can steer the optimization process away from the minimum and might even cause catastrophic forgetting [38] of previously learned concepts.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [38]

      Not checked in the free run.

  • May combine two claims.

    Interpretationcitation
    § 1) Local Batch Size:

    Partial participation reduces the convergence speed of STC by causing the clients' residuals to go out of sync and increasing gradient staleness.

    “On the other hand, partial participation reduces the convergence speed of STC by causing the clients residuals to go out sync and increasing the gradient staleness [25].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [25]

      Not checked in the free run.

  • The quote only partly states this claim.

    Interpretationown data
    § A. Momentum in Federated Optimization

    For STC and federated averaging, whether momentum is beneficial or harmful depends on three parameters of the learning environment.

    “We can make out three different parameters of the learning environment that determine whether momentum is beneficial or harmful to the performance of STC.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Study design and conductreasoning
    § A. Preliminary Experiments

    To make the results comparable, all compression methods use the same learning rate and batch size.

    “To make the results comparable, all compression methods use the same learning rate and batch size.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Premisecitation
    § C. Weight Update Caching for Partial Client Participation

    The downlink typically is cheaper and has far higher bandwidth than the uplink.

    “This is usually tolerable as the downlink typically is cheaper and has far higher bandwidth than the uplink, as already noted in [10] and [19].”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites [10]

      Not checked in the free run.

    • cites [19]

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

    Interpretationreasoning
    § II. CHALLENGES OF THE FEDERATED LEARNING ENVIRONMENT

    The quality of the collaboratively learned model is determined by the combined available data of all clients, so collaborative learning environments have a natural tendency to grow.

    “Furthermore, as the quality of the collaboratively learned model is determined by the combined available data of all clients, collaborative learning environments will have a natural tendency to grow.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Study design and conductreasoning
    § D. Communication Efficiency

    To have a meaningful comparison, the communication efficiency evaluation is performed on an i.i.d. environment with moderate batch size, which favors federated averaging and signSGD.

    “To still have a meaningful comparison, we, therefore, choose to evaluate this time on an i.i.d.environment where every client holds ten different classes and uses a moderate batch size of 20 during training.This setup favors federated averaging and signSGD to the maximum degree possible!”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Study design and conductnone given
    § 1) Local Batch Size:

    To isolate the effects of reduced participation, the absolute number of participating clients and local batch sizes are kept constant while varying total number of clients.

    “To isolate the effects of reduced participation, we keep the absolute number of participating clients and the local batch sizes at constant values of 5 and 40, respectively, throughout all experiments and vary only the total number of clients (and thus the relative participation η).”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Study design and conductnone given
    § Compression Methods:

    All training related hyperparameters are kept constant for the different compression methods.

    “During our experiments, we keep all training related hyperparameters constant for the different compression methods.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Study design and conductreasoning
    § Compression Methods:

    All methods are given the same budget of training iterations in the experiments.

    “To be able to compare the different methods in a fair way, all methods are given the same budged of training iterations in the following experiments”
    Quote in the paperSubstantial claim
39 of 39

  • The authors state this as an assumption.

    The size of the model and the number of training iterations are fixed.

    “If we assume the size of the model and number of training iterations to be fixed (e.g., because we want to achieve a certain accuracy on a given task)”

  • The authors state this as an assumption.

    The sparsity pattern of the weight updates is random.

    “Assuming a random sparsity pattern we know that for big values of |W | and k = p|W |, the distances are approximately geometrically distributed with success probability equal to the sparsity rate p.”

  • The authors state this as an assumption.

    The objective function is L-smooth, μ-strongly convex, and has bounded gradients.

    “for any L-smooth, μ-strongly convex objective function f with bounded gradients EW 2 ≤ G 2”

  • The authors state this as an assumption.

    A public i.i.d. data set shared by all clients is generally not available.

    “the most crucial one being that we generally cannot assume the availability of such a public data set.”

  • The paper relies on this without support.

    In federated learning, typically only a fraction of the entire client population participates in any particular communication round.

    “However, as elaborated in Section II, in federated learning, typically only a fraction of the entire client population will participate in any particular communication round.”

  • The paper relies on this without support.

    i.i.d. distribution of client data typically cannot be found in federated learning.

    “which typically cannot be found in federated learning”

  • The paper relies on this without support.

    Federated learning comes at the cost of a significant communication overhead during training.

    “This form of privacy-preserving collaborative learning, however, comes at the cost of a significant communication overhead during training.”
7 of 7

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • Figure 5 has conflicting iteration counts

    Minor

    The discussion says the Figure 5 experiment ran for 10 000 iterations, whereas its caption says 16 000 iterations.

  • Figure 8 participation range conflicts with the reported extreme case

    Minor

    The discussion presents 5 out of 400 participating clients as an extreme case in its account of Figure 8, but the Figure 8 caption says its range ends at 5 out of 100 clients.

2 of 2

Cites

Flagged first, then by citations

Research works printed but not linked: 2

  1. lecun, 1998, The MNIST Database of Handwritten Digits
  2. krizhevsky, 2014, CIFAR-10 Dataset

Cited by

Sorted by citations

Related

Authors
Sattler, F., Wiedemann, S., Muller, K.-R., & Samek, W.
Affiliations
  • Fraunhofer Institute for Telecommunications, Heinrich Hertz Institute (DE)
  • Technische Universität Berlin, Berlin, Germany
ORCID
Venue
IEEE Transactions on Neural Networks and Learning Systems
Publisher
Institute of Electrical and Electronics Engineers (IEEE)
Published
2020-09
Volume
31
Issue
9
Pages
3400-3413
ISSN
  • 2162-237X (ISSN-L)
  • 2162-2388
DOI
10.1109/tnnls.2019.2944481
OpenAlex
W2918777711
Semantic Scholar
29081e5c63d24fa8952e938dba68956cd47ac81d
Cited by
1,725
References
39
Citations by year
2026: 161 · 2025: 273 · 2024: 313 · 2023: 344 · 2022: 288 · 2021: 229 · 2020: 100 · 2019: 17
FWCI
72.33
Type
journal article
Language
en
Open access
Licence
Locations
  • IEEE Transactions on Neural Networks and Learning Systems (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • Publikationsdatenbank der Fraunhofer-Gesellschaft (Fraunhofer-Gesellschaft) (submittedVersion, other-oa)
  • Fraunhofer-Publica (Fraunhofer-Gesellschaft) (submittedVersion)
  • arXiv (Cornell University)
Funders
  • Fraunhofer Society through the MPI-FhG Collaboration Project “Theory & Practice for Reduced Learning Machines,”
  • German Ministry for Education and Research as Berlin Big Data Center (01IS14013A)
  • Berlin Center for Machine Learning (01IS18037I)
  • DFG (EXC 2046/1, 390685689)
  • Information & Communications Technology Planning & Evaluation
  • Korea Government (2017-0-00451)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Privacy-Preserving Technologies in Data
  • Physical Sciences › Computer Science › Artificial Intelligence › Stochastic Gradient Optimization Techniques
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Data Classification
Keywords
federated learning, non-IID data, communication efficiency, gradient sparsification, FedAvg, privacy-preserving collaborative learning

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-09.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-09.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review