CoChat

Federated Variance-Reduced Stochastic Gradient Descent With Robustness to Byzantine Attacks

Open access
Wu, Zhaoxian, Ling, Qing, Chen, Tianyi, Giannakis, Georgios B. · 2020 · IEEE Transactions on Signal Processing

The DOI matches its Crossref record; 47 of the 51 references it cites were found in a registry or an index.

Abstract

This paper deals with distributed finite-sum optimization for learning over multiple workers in the presence of malicious Byzantine attacks. Most resilient approaches so far combine stochastic gradient descent (SGD) with different robust aggregation rules. However, the sizeable SGD-induced stochastic gradient noise challenges discerning malicious messages sent by the Byzantine attackers from noisy stochastic gradients sent by the `honest' workers. This motivates reducing the variance of stochastic gradients as a means of robustifying SGD. To this end, a novel Byzantine attack resilient distributed (Byrd-) SAGA approach is introduced for federated learning tasks involving multiple workers. Rather than the mean employed by distributed SAGA, the novel Byrd-SAGA relies on the geometric median to aggregate the corrected stochastic gradients sent by the workers. When less than half of the workers are Byzantine attackers, Byrd-SAGA attains provably linear convergence to a neighborhood of the optimal solution, with the asymptotic learning error determined by the number of Byzantine workers. Numerical tests corroborate the robustness to various Byzantine attacks, as well as the merits of Byrd-SAGA over Byzantine attack resilient distributed SGD.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1109/tsp.2020.3012952

Cites

47 of 47 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 47 references

Research works printed but not linked: 2

  1. tang, 0, D2: Decentralized training over decentralized data, Proc Int Conf Mach Learn, 4855
  2. yang, 2019, Federated Learning

Cited by

Metadata

Authors
Wu, Z., Ling, Q., Chen, T., & Giannakis, G. B.
Affiliations
  • Sun Yat-sen University (CN)
  • Rensselaer Polytechnic Institute (US)
  • University of Minnesota (US)
Venue
IEEE Transactions on Signal Processing
Published
2020
Volume
68
Pages
4583-4596
ISSN
  • 1053-587X (ISSN-L)
  • 1941-0476
OpenAlex
W3046449784
Cited by
226
References
51
Citations by year
2026: 22 · 2025: 45 · 2024: 41 · 2023: 40 · 2022: 38 · 2021: 32 · 2020: 8
FWCI
11.60
Type
journal article
Language
en
Open access
Locations
  • IEEE Transactions on Signal Processing (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Funders
  • NSF (61573331, 61973324, 1509040, 1508993, 1711471, 1901134)
  • Fundamental Research Funds for the Central Universities
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Stochastic Gradient Optimization Techniques
  • Physical Sciences › Computer Science › Artificial Intelligence › Privacy-Preserving Technologies in Data
  • Physical Sciences › Engineering › Control and Systems Engineering › Distributed Control Multi-Agent Systems
Keywords
federated learning, Byzantine attacks, variance reduction, stochastic gradient descent, sagas, geometric median, robust aggregation, distributed optimization, finite-sum optimization, linear convergence, malicious workers

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.