Machine Learning at the Wireless Edge: Distributed Stochastic Gradient Descent Over-the-Air
Open accessThe DOI matches its Crossref record; 7 of the 7 references it cites were found in a registry or an index.
Abstract
We study collaborative machine learning at the wireless edge, where power and bandwidth-limited devices (workers), with limited local datasets, implement distributed stochastic gradient descent (DSGD) over-the-air with the help of a remote parameter server (PS). We consider a wireless multiple access channel (MAC) from the workers to the PS for communicating the local gradient estimates. We first introduce a digital DSGD (D-DSGD) scheme, assuming that the workers operate on the boundary of the MAC capacity region at each iteration of the DSGD algorithm, and digitize their estimates within the bit budget allowed by the employed power allocation. We then introduce an analog scheme, called A-DSGD, motivated by the additive nature of the wireless MAC, where the workers send their gradient estimates over the MAC through the available channel bandwidth without employing any digital code. Numerical results show that A-DSGD converges much faster than D-DSGD. The improvement is particularly compelling at low power and low bandwidth regimes. We also observe that the performance of A-DSGD improves with the number of workers, while D-DSGD deteriorates, limiting the ability of the latter in harnessing the computation power of many edge devices.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
Communication-Efficient Learning of Deep Networks from Decentralized Data
McMahan, H. B., Moore, E., Ramage, D., Hampson, S., & Arcas, B. A. Y. (2016) · arXiv · doi:10.48550/arxiv.1602.05629 · Cited by 14,215
Deep Learning with Limited Numerical Precision
Gupta, S., Agrawal, A., Gopalakrishnan, K., & Narayanan, P. (2015) · arXiv · doi:10.48550/arxiv.1502.02551 · Cited by 1,364 · cited 1×
“Quantization methods implement lossy compression of the gradients by quantizing each of their entries to a finitebit low precision value [1], [3], [5].”
MethodDeep Gradient Compression: Reducing the Communication Bandwidth for Distributed Training
Lin, Y., Han, S., Mao, H., Wang, Y., & Dally, W. J. (2017) · arXiv · doi:10.48550/arxiv.1712.01887 · Cited by 1,134 · cited 4×
“Another approach is to reduce the frequency of communication from the workers by allowing local parameter updates [3], [32]–[35].”
BackgroundMethodInfluentialFederated Learning via Over-the-Air Computation
Yang, K., Jiang, T., Shi, Y., & Ding, Z. (2020) · IEEE Transactions on Wireless Communications · doi:10.1109/twc.2019.2961673 · Cited by 1,084 · cited 3×
“While our work can be easily extended to the fading model, the distinctive contributions of our work with respect to the concurrent works of [40] and [41] are i) the introduction of a purely digital separate gradient compression and communication scheme; ii) the consideration of a bandwidth-constrained channel, which…”
BackgroundSparse Communication for Distributed Gradient Descent
Aji, A. F., & Heafield, K. (2017) · Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d17-1045 · Cited by 585
Sparse Binary Compression: Towards Distributed Deep Learning with minimal Communication
Sattler, F., Wiedemann, S., Muller, K.-R., & Samek, W. (2019) · 2019 International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.2019.8852172 · Cited by 240
Cited by
Edge Intelligence: Paving the Last Mile of Artificial Intelligence With Edge Computing
Zhou, Z., Chen, X., Li, E., Zeng, L., Luo, K., & Zhang, J. (2019) · Proceedings of the IEEE · doi:10.1109/jproc.2019.2918951 · Cited by 2,455
Energy Efficient Federated Learning Over Wireless Communication Networks
Yang, Z., Chen, M., Saad, W., Hong, C. S., & Shikh-Bahaei, M. (2021) · IEEE Transactions on Wireless Communications · doi:10.1109/twc.2020.3037554 · Cited by 1,153
Broadband Analog Aggregation for Low-Latency Federated Edge Learning
Zhu, G., Wang, Y., & Huang, K. (2020) · IEEE Transactions on Wireless Communications · doi:10.1109/twc.2019.2946245 · Cited by 761
Federated Learning Over Wireless Fading Channels
Amiri, M. M., & Gunduz, D. (2020) · IEEE Transactions on Wireless Communications · doi:10.1109/twc.2020.2974748 · Cited by 625
“Finally, we propose our scheme, built upon our previous work [25], referred to as the compressed analog DSGD (CA-DSGD) scheme.”
BackgroundMethodResultInfluentialSelf-citationToward an Intelligent Edge: Wireless Communication Meets Machine Learning
Zhu, G., Liu, D., Du, Y., You, C., Zhang, J., & Huang, K. (2020) · IEEE Communications Magazine · doi:10.1109/mcom.001.1900103 · Cited by 606
Metadata
- Authors
- Amiri, M. M., & Gunduz, D.
- Venue
- 2019 IEEE International Symposium on Information Theory (ISIT)
- Publisher
- IEEE
- Published
- 2019-07
- Pages
- 1432-1436
- OpenAlex
- W2975064142
- Cited by
- 109
- References
- 7
- FWCI
- 19.80
- Type
- conference paper
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.