Federated Learning Over Wireless Fading Channels
Open accessThe DOI matches its Crossref record; 41 of the 44 references it cites were found in a registry or an index.
Abstract
We study federated machine learning at the wireless network edge, where limited power wireless devices, each with its own dataset, build a joint model with the help of a remote parameter server (PS). We consider a bandwidth-limited fading multiple access channel (MAC) from the wireless devices to the PS, and propose various techniques to implement distributed stochastic gradient descent (DSGD) over this shared noisy wireless channel. We first propose a digital DSGD (D-DSGD) scheme, in which one device is selected opportunistically for transmission at each iteration based on the channel conditions; the scheduled device quantizes its gradient estimate to a finite number of bits imposed by the channel condition, and transmits these bits to the PS in a reliable manner. Next, motivated by the additive nature of the wireless MAC, we propose a novel analog communication scheme, referred to as the compressed analog DSGD (CA-DSGD), where the devices first sparsify their gradient estimates while accumulating error from previous iterations, and project the resultant sparse vector into a low-dimensional vector for bandwidth reduction. We also design a power allocation scheme to align the received gradient vectors at the PS in an efficient manner. Numerical results show that D-DSGD outperforms other digital approaches in the literature; however, in general the proposed CA-DSGD algorithm converges faster than the D-DSGD scheme, and reaches a higher level of accuracy. We have observed that the gap between the analog and digital schemes increases when the datasets of devices are not independent and identically distributed (i.i.d.). Furthermore, the performance of the CA-DSGD scheme is shown to be robust against imperfect channel state information (CSI) at the devices. Overall these results show clear advantages for the proposed analog over-the-air DSGD scheme, which suggests that learning and communication algorithms should be designed jointly to achieve the best end-to-end performance in machine learning applications at the wireless edge.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“The convergence behavior demonstrates that the noisy iterates typically introduce non-negligible optimality gap in various FL algorithms, e.g., vanilla gradient method [174], quantized gradient method [175], sparsified gradient method [173], and operator splitting method [87].”
From Edge artificial intelligence for 6g: vision, enabling technologies, and applications (2022) · cites this paper as [173]
In the citing paper
“The optimality gap can be further controlled by transmit power allocation [41], [173], [176], model aggregation receiver beamforming design [31], [177], [178], and device scheduling [31], [178], [179].”
From Edge artificial intelligence for 6g: vision, enabling technologies, and applications (2022) · cites this paper as [173]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
Communication-Efficient Learning of Deep Networks from Decentralized Data
McMahan, H. B., Moore, E., Ramage, D., Hampson, S., & Arcas, B. A. Y. (2016) · arXiv · doi:10.48550/arxiv.1602.05629 · Cited by 14,215
Fundamentals of Wireless Communication
Tse, D., & Viswanath, P. (2005) · doi:10.1017/cbo9780511807213 · Cited by 13,162 · cited 1×
“For a transmit power P m ∗ ( t ) ( t ) , the capacity of this parallel Gaussian channel is the result of the following optimization problem [32, Section 5.4.6]: The optimization problem in (9) is solved through waterfilling, and the optimal power allocation is given by (cid:12)(cid:12) where ζ is…”
BackgroundInfluentialChannel Coding Rate in the Finite Blocklength Regime
Polyanskiy, Y., Poor, H. V., & Verdu, S. (2010) · IEEE Transactions on Information Theory · doi:10.1109/tit.2010.2043769 · Cited by 3,885 · cited 1×
“In practice, this would result in a higher error probability or reduced transmission rate [33].”
BackgroundFederated Learning: Strategies for Improving Communication Efficiency
Konečný, J., McMahan, H. B., Yu, F. X., Richtárik, P., Suresh, A. T., & Bacon, D. (2016) · arXiv · doi:10.48550/arxiv.1610.05492 · Cited by 3,766
Message-passing algorithms for compressed sensing
Donoho, D. L., Maleki, A., & Montanari, A. (2009) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.0909892106 · Cited by 2,416 · cited 1×
“For this, using its knowledge of matrix A and the CSI, PS employs the approximate message passing (AMP) algorithm [29].”
MethodDoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients
Zhou, S., Wu, Y., Ni, Z., Zhou, X., Wen, H., & Zou, Y. (2016) · arXiv · doi:10.48550/arxiv.1606.06160 · Cited by 1,690
Deep Learning with Limited Numerical Precision
Gupta, S., Agrawal, A., Gopalakrishnan, K., & Narayanan, P. (2015) · International Conference on Machine Learning · Cited by 1,590
Deep Learning with Limited Numerical Precision
Gupta, S., Agrawal, A., Gopalakrishnan, K., & Narayanan, P. (2015) · arXiv · doi:10.48550/arxiv.1502.02551 · Cited by 1,364
Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Training
Lin, Y., Han, S., Mao, H., Wang, Y., & Dally, W. J. (2017) · arXiv · doi:10.48550/arxiv.1712.01887 · Cited by 1,134
Federated Learning via Over-the-Air Computation
Yang, K., Jiang, T., Shi, Y., & Ding, Z. (2018) · arXiv · doi:10.48550/arxiv.1812.11750 · Cited by 1,090 · cited 3×
“Then we studied the alternative analog transmission approach, cosidered recently in [25]–[27], which does not employ quantization or channel coding, and exploits the superposition property of the wireless MAC, rather than orthogonalizing the transmissions from different devices.”
MethodFederated Learning via Over-the-Air Computation
Yang, K., Jiang, T., Shi, Y., & Ding, Z. (2020) · IEEE Transactions on Wireless Communications · doi:10.1109/twc.2019.2961673 · Cited by 1,084 · cited 3×
“Then we studied the alternative analog transmission approach, cosidered recently in [25]–[27], which does not employ quantization or channel coding, and exploits the superposition property of the wireless MAC, rather than orthogonalizing the transmissions from different devices.”
MethodTernGrad: Ternary Gradients to Reduce Communication in Distributed Deep Learning
Wen, W., Xu, C., Yan, F., Wu, C., Wang, Y., Chen, Y., & Li, H. H. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1705.07878 · Cited by 622
Sparse Communication for Distributed Gradient Descent
Aji, A. F., & Heafield, K. (2017) · Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d17-1045 · Cited by 585
Machine Learning at the Wireless Edge: Distributed Stochastic Gradient Descent Over-the-Air
Mohammadi Amiri, M., & Gunduz, D. (2020) · IEEE Transactions on Signal Processing · doi:10.1109/tsp.2020.2981904 · Cited by 548
Self-citationsignSGD: Compressed Optimisation for Non-Convex Problems
Bernstein, J., Wang, Y.-X., Azizzadenesheli, K., & Anandkumar, A. (2018) · arXiv · doi:10.48550/arxiv.1802.04434 · Cited by 541 · cited 8×
“For the MNIST classification task, it is shown that the proposed digital approach D-DSGD outperforms digital schemes that employ QSGD [8] or SignSGD [12] for gradient compression.”
MethodInfluentialThe Convergence of Sparsified Gradient MethodsPreprint
Alistarh, D., Hoefler, T., Johansson, M., Khirirat, S., Konstantinov, N., & Renggli, C. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1809.10505 · Cited by 525
Scalable distributed DNN training using commodity GPU cloud computing
Strom, N. (2015) · Interspeech 2015 · doi:10.21437/interspeech.2015-354 · Cited by 521
Seide, F., Fu, H., Droppo, J., Li, G., & Yu, D. (2014) · Interspeech 2014 · doi:10.21437/interspeech.2014-274 · Cited by 443 · cited 1×
“error accumulation [7], where the accumulated error vector at device m until iteration t is denoted by Δm(t − 1) ∈ R, where we set Δm(0) = 0, ∀m ∈ [M ].”
BackgroundLAG: Lazily Aggregated Gradient for Communication-Efficient Distributed LearningPreprint
Chen, T., Giannakis, G. B., Sun, T., & Yin, W. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1805.09965 · Cited by 329 · cited 1×
“Therefore, there have been significant research efforts to reduce the communication requirements in FL [1]–[24].”
BackgroundSparse Binary Compression: Towards Distributed Deep Learning with minimal Communication
Sattler, F., Wiedemann, S., Muller, K.-R., & Samek, W. (2019) · 2019 International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.2019.8852172 · Cited by 240 · cited 3×
“The scheduled device quantizes its gradient estimate to a finite number of bits using the gradient compression scheme in [18] while accumulating the error from previous iterations (this will be clarified later), and employs a channel code to transmit the bits over the available bandwidth-limited channel to the PS.”
MethodDon't Use Large Mini-Batches, Use Local SGD
Lin, T., Stich, S. U., Patel, K. K., & Jaggi, M. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1808.07217 · Cited by 193
Local SGD Converges Fast and Communicates Little
Stich, S. U. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1805.09767 · Cited by 191
ATOMO: Communication-efficient Learning via Atomic Sparsification
Wang, H., Sievert, S., Charles, Z., Liu, S., Wright, S. J., & Papailiopoulos, D. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1806.04090 · Cited by 189
Machine Learning at the Wireless Edge: Distributed Stochastic Gradient Descent Over-the-Air
Amiri, M. M., & Gunduz, D. (2019) · 2019 IEEE International Symposium on Information Theory (ISIT) · doi:10.1109/isit.2019.8849334 · Cited by 109 · cited 7×
“Finally, we propose our scheme, built upon our previous work [25], referred to as the compressed analog DSGD (CA-DSGD) scheme.”
BackgroundMethodResultInfluentialSelf-citationmeProp: Sparsified Back Propagation for Accelerated Deep Learning with Reduced Overfitting
Sun, X., Ren, X., Ma, S., & Wang, H. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1706.06197 · Cited by 106
SoftCast: one-size-fits-all wireless video
Jakubczak, S., & Katabi, D. (2010) · ACM SIGCOMM Conference (SIGCOMM) · doi:10.1145/1851182.1851257 · Cited by 104 · cited 1×
“Analog transmission has been well studied for image/ video multicasting over wireless channels in recent years [29]–[31], and here we employ the random projection technique proposed in [31] for image transmission over a bandwidth limited wireless channel.”
MethodSparCML: high-performance sparse communication for machine learning
Renggli, C., Ashkboos, S., Aghagolzadeh, M., Alistarh, D., & Hoefler, T. (2019) · Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis · doi:10.1145/3295500.3356222 · Cited by 101
Smith, V., Chiang, C.-K., Sanjabi, M., & Talwalkar, A. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1705.10467 · Cited by 101
Randomized distributed mean estimation: Accuracy vs. communication
Konečný, J., & Richtárik, P. (2018) · Frontiers in Applied Mathematics and Statistics · doi:10.3389/fams.2018.00062 · Cited by 69
Over-the-Air Machine Learning at the Wireless Edge
Amiri, M. M., & Gündüz, D. (2019) · 2019 IEEE 20th International Workshop on Signal Processing Advances in Wireless Communications (SPAWC) · doi:10.1109/spawc.2019.8815402 · Cited by 60 · cited 1×
“Here, we extend our previous works [25], [28], and study DSGD over a wireless fading MAC.”
MethodSelf-citationSoftCast: one-size-fits-all wireless video
Jakubczak, S., & Katabi, D. (2010) · ACM SIGCOMM Computer Communication Review · doi:10.1145/1851275.1851257 · Cited by 51
Variance-based Gradient Compression for Efficient Distributed Deep Learning
Tsuzuku, Y., Imachi, H., & Akiba, T. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1802.06058 · Cited by 48
Low-Latency Broadband Analog Aggregation for Federated Edge Learning.Preprint
Zhu, G., Wang, Y., & Huang, K. (2018) · arXiv (Cornell University) · Cited by 43
Power Distortion Optimization for Uncoded Linear Transformed Transmission of Images and Videos
Xiong, R., Zhang, J., Wu, F., Xu, J., & Gao, W. (2016) · IEEE Transactions on Image Processing · doi:10.1109/tip.2016.2621478 · Cited by 35
SparCML: High-Performance Sparse Communication for Machine LearningPreprint
Renggli, C., Ashkboos, S., Aghagolzadeh, M., Alistarh, D., & Hoefler, T. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1802.08021 · Cited by 31
Broadband Analog Aggregation for Low-Latency Federated Edge Learning (Extended Version)
Zhu, G., Wang, Y., & Huang, K. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1812.11494 · Cited by 29 · cited 5×
“In [26] the authors consider a fading MAC, and also apply analog transmission, where each entry of a gradient vector at each of the devices is scheduled for transmission depending on the corresponding channel condition.”
BackgroundMethodInfluentialSparseCast: Hybrid Digital-Analog Wireless Image Transmission Exploiting Frequency-Domain Sparsity
Tung, T.-Y., & Gündüz, D. (2018) · IEEE Communications Letters · doi:10.1109/lcomm.2018.2877316 · Cited by 25 · cited 1×
“Analog transmission has been well studied for image/ video multicasting over wireless channels in recent years [29]–[31], and here we employ the random projection technique proposed in [31] for image transmission over a bandwidth limited wireless channel.”
MethodRunning sparse and low-precision neural network: When algorithm meets hardware
Li, B., Wen, W., Mao, J., Li, S., Chen, Y., & Li, H. (2018) · Asia and South Pacific Design Automation Conference (ASP-DAC) · doi:10.1109/aspdac.2018.8297378 · Cited by 18
Randomized Distributed Mean Estimation: Accuracy vs CommunicationPreprint
Konečný, J., & Richtárik, P. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.07555 · Cited by 8
Running sparse and low-precision neural network: when algorithm meets hardware
Li, B., Wen, W., Mao, J., Li, S., Chen, Y., & Li, H. H. (2018) · Asia and South Pacific Design Automation Conference (ASP-DAC) · doi:10.5555/3201607.3201733 · Cited by 8
Research works printed but not linked: 3
- lecun, 1998, The MNIST Database of Handwritten Digits
- alistarh, 2017, QSGD: Communication-efficient SGD via randomized quantization and encoding, Proc NIPS, 1709
- mcmahan, 2017, Federated learning Collaborative machine learning without centralized training data
Cited by
Federated Learning in Mobile Edge Networks: A Comprehensive Survey
Lim, W. Y. B., Luong, N. C., Hoang, D. T., Jiao, Y., Liang, Y.-C., Yang, Q., Niyato, D., & Miao, C. (2020) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2020.2986024 · Cited by 2,565
“As an extension, the authors in [129] introduce error accumulation and gradient sparsification in addition to over-the-air computation.”
BackgroundMethodA survey on federated learning
Zhang, C., Xie, Y., Bai, H., Yu, B., Li, W., & Gao, Y. (2021) · Knowledge-Based Systems · doi:10.1016/j.knosys.2021.106775 · Cited by 1,899
Federated Learning for Internet of Things: Recent Advances, Taxonomy, and Open Challenges
Khan, L. U., Saad, W., Han, Z., Hossain, E., & Hong, C. S. (2021) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2021.3090430 · Cited by 892
Edge artificial intelligence for 6g: vision, enabling technologies, and applications
Letaief, K. B., Shi, Y., Lu, J., & Lu, J. (2022) · IEEE Journal on Selected Areas in Communications · doi:10.1109/jsac.2021.3126076 · Cited by 829
“The convergence behavior demonstrates that the noisy iterates typically introduce non-negligible optimality gap in various FL algorithms, e.g., vanilla gradient method [174], quantized gradient method [175], sparsified gradient method [173], and operator splitting method [87].”
BackgroundMethodBeyond Transmitting Bits: Context, Semantics, and Task-Oriented Communications
Gunduz, D., Qin, Z., Aguerri, I. E., Dhillon, H. S., Yang, Z., Yener, A., Wong, K. K., & Chae, C.-B. (2023) · IEEE Journal on Selected Areas in Communications · doi:10.1109/jsac.2022.3223408 · Cited by 776
“This ‘over-the-air computation (OAC)’ approach has received significant interest in recent years thanks to its bandwidth efficiency [263], [264], [268],”
Metadata
- Authors
- Amiri, M. M., & Gunduz, D.
- Affiliations
- Princeton University (US)
- Venue
- IEEE Transactions on Wireless Communications
- Published
- 2020-05
- Volume
- 19
- Issue
- 5
- Pages
- 3546-3557
- ISSN
- 1536-1276 (ISSN-L)
- 1558-2248
- OpenAlex
- W2963647784
- Cited by
- 625
- References
- 44
- Citations by year
- 2026: 47 · 2025: 66 · 2024: 93 · 2023: 111 · 2022: 129 · 2021: 126 · 2020: 47
- FWCI
- 40.56
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- IEEE Transactions on Wireless Communications (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- IRIS UNIMORE (University of Modena and Reggio Emilia) (submittedVersion)
- IRIS UNIMORE (University of Modena and Reggio Emilia) (submittedVersion, cc-by)
- Spiral (Imperial College London) (submittedVersion)
- arXiv (Cornell University)
- Funders
- European Research Council (ERC) Starting Grant BEACON (677854)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Privacy-Preserving Technologies in Data
- Physical Sciences › Computer Science › Computer Networks and Communications › Distributed Sensor Networks and Detection Algorithms
- Physical Sciences › Computer Science › Artificial Intelligence › Stochastic Gradient Optimization Techniques
- Keywords
- federated learning, wireless edge computing, fading multiple access channel, distributed stochastic gradient descent, over-the-air computation, gradient quantization, gradient sparsification, power allocation, bandwidth-limited channels, non-IID data, imperfect channel state information, parameter server
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.