UVeQFed: Universal Vector Quantization for Federated Learning
Open accessThe DOI matches its Crossref record; 64 of the 65 references it cites were found in a registry or an index.
Abstract
Traditional deep learning models are trained at a centralized server using data samples collected from users. Such data samples often include private information, which the users may not be willing to share. Federated learning (FL) is an emerging approach to train such learning models without requiring the users to share their data. FL consists of an iterative procedure, where in each iteration the users train a copy of the learning model locally. The server then collects the individual updates and aggregates them into a global model. A major challenge that arises in this method is the need of each user to repeatedly transmit its learned model over the throughput limited uplink channel. In this work, we tackle this challenge using tools from quantization theory. In particular, we identify the unique characteristics associated with conveying trained models over rate-constrained channels, and propose a suitable quantization scheme for such settings, referred to as universal vector quantization for FL (UVeQFed). We show that combining universal vector quantization methods with FL yields a decentralized training system in which the compression of the trained models induces only a minimum distortion. We then theoretically analyze the distortion, showing that it vanishes as the number of users grows. We also characterize how models trained with conventional federated averaging combined with UVeQFed converge to the model which minimizes the loss function. Our numerical results demonstrate the gains of UVeQFed over previously proposed methods in terms of both distortion induced in quantization and accuracy of the resulting aggregated model.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“Scalar quantization is a typical way to implement lossy compression for the high-dimensional gradient vectors by quantizing each of their entries to a finite-bit low precision value [123]- [125], which was further improved by the recent proposal of vector quantization [126], [127].”
From Edge artificial intelligence for 6g: vision, enabling technologies, and applications (2022) · cites this paper as [127]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
LeCun, Y., Bengio, Y., & Hinton, G. (2015) · Nature · doi:10.1038/nature14539 · Cited by 84,126
Elements of Information Theory
Cover, T. M., & Thomas, J. A. (2001) · Wiley eBooks · doi:10.1002/0471200611 · Cited by 36,808
Communication-Efficient Learning of Deep Networks from Decentralized Data
McMahan, H. B., Moore, E., Ramage, D., Hampson, S., & Arcas, B. A. Y. (2016) · arXiv · doi:10.48550/arxiv.1602.05629 · Cited by 14,215
Advances and Open Problems in Federated Learning
Kairouz, P., & McMahan, H. B. (2021) · Foundations and Trends in Machine Learning · doi:10.1561/2200000083 · Cited by 7,772
Federated learning: Challenges, methods, and future directions
Li, T., Sahu, A. K., Talwalkar, A., & Smith, V. (2020) · IEEE Signal Processing Magazine · doi:10.1109/msp.2020.2975749 · Cited by 6,210 · cited 3×
“One of the major challenges of FL is the transfer of a large number of updated model parameters over the uplink communication channel from the users to the server, whose throughput is typically constrained [5], [7]–[9].”
BackgroundLarge Scale Distributed Deep Networks
Dean, J. B., Corrado, G. S., Monga, R., Chen, K., Devin, M., Mao, M., Ranzato, M., Senior, A., Tucker, P. A., Yang, K., et al. (2012) · neural information processing systems · Cited by 4,489
Federated Learning: Strategies for Improving Communication Efficiency
Konečný, J., McMahan, H. B., Yu, F. X., Richtárik, P., Suresh, A. T., & Bacon, D. (2016) · arXiv · doi:10.48550/arxiv.1610.05492 · Cited by 3,766
Sphere Packings, Lattices and Groups.
Coxeter, H. S. M., Conway, J. H., & Sloane, N. J. A. (1989) · The American Mathematical Monthly · doi:10.2307/2323992 · Cited by 3,377
The rate-distortion function for source coding with side information at the decoder
Wyner, A., & Ziv, J. (1976) · IEEE Transactions on Information Theory · doi:10.1109/tit.1976.1055508 · Cited by 3,023
Towards Federated Learning at Scale: System Design
Bonawitz, K., Eichner, H., Grieskamp, W., Huba, D., Ingerman, A., Ivanov, V., Kiddon, C., Konečný, J., Mazzocchi, S., McMahan, H. B., et al. (2019) · arXiv · doi:10.48550/arxiv.1902.01046 · Cited by 2,334
The Roadmap to 6G: AI Empowered Wireless Networks
Letaief, K. B., Chen, W., Shi, Y., Zhang, J., & Zhang, Y.-J. A. (2019) · IEEE Communications Magazine · doi:10.1109/mcom.2019.1900271 · Cited by 2,207
Gray, R. M., & Neuhoff, D. L. (1998) · IEEE Transactions on Information Theory · doi:10.1109/18.720541 · Cited by 1,642
On the Convergence of FedAvg on Non-IID Data
Li, X., Huang, K., Yang, W., Wang, S., & Zhang, Z. (2019) · arXiv · doi:10.48550/arxiv.1907.02189 · Cited by 1,575
A Joint Learning and Communications Framework for Federated Learning over Wireless Networks
Chen, M., Yang, Z., Saad, W., Yin, C., Poor, H. V., & Cui, S. (2019) · arXiv · doi:10.48550/arxiv.1909.07972 · Cited by 1,547
Self-citationA Joint Learning and Communications Framework for Federated Learning Over Wireless Networks
Chen, M., Yang, Z., Saad, W., Yin, C., Poor, H. V., & Cui, S. (2021) · IEEE Transactions on Wireless Communications · doi:10.1109/twc.2020.3024629 · Cited by 1,547
Self-citationDeep Learning With Edge Computing: A Review
Chen, J., & Ran, X. (2019) · Proceedings of the IEEE · doi:10.1109/jproc.2019.2921977 · Cited by 1,540
Sphere Packings, Lattices and Groups
Conway, J. H., & Sloane, N. J. A. (1988) · Grundlehren der mathematischen Wissenschaften · doi:10.1007/978-1-4757-2016-7 · Cited by 1,532
Federated Learning of Deep Networks using Model AveragingPreprint
McMahan, H. B., Moore, E., Ramage, D., & y Arcas, B. A. (2016) · arXiv (Cornell University) · Cited by 1,288
Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Training
Lin, Y., Han, S., Mao, H., Wang, Y., & Dally, W. J. (2017) · arXiv · doi:10.48550/arxiv.1712.01887 · Cited by 1,134 · cited 1×
“Sparsifying masks for compressing the gradients were proposed in [13]–[15].”
BackgroundArtificial Neural Networks-Based Machine Learning for Wireless Networks: A Tutorial
Chen, M., Challita, U., Saad, W., Yin, C., & Debbah, M. (2019) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2019.2926625 · Cited by 1,117
Self-citationFederated Learning via Over-the-Air Computation
Yang, K., Jiang, T., Shi, Y., & Ding, Z. (2020) · IEEE Transactions on Wireless Communications · doi:10.1109/twc.2019.2961673 · Cited by 1,084 · cited 1×
“This property of wireless communications is known to affect the design of deep learning systems operating over such channels [41]–[47].”
BackgroundElements of Information Theory
(2013) · Wireless Communications · doi:10.1002/9781118576618.ch7 · Cited by 999
Kang, J., Xiong, Z., Niyato, D., Xie, S., & Zhang, J. (2019) · IEEE Internet of Things Journal · doi:10.1109/jiot.2019.2940820 · Cited by 953
QSGD: Communication-Efficient SGD via Gradient Quantization and Encoding
Alistarh, D., Grubic, D., Li, J., Tomioka, R., & Vojnović, M. (2016) · London School of Economics and Political Science Research Online (London School of Economics and Political Science) · doi:10.48550/arxiv.1610.02132 · Cited by 859
Scheduling Policies for Federated Learning in Wireless Networks
Yang, H. H., Liu, Z., Quek, T. Q. S., & Poor, H. V. (2020) · IEEE Transactions on Communications · doi:10.1109/tcomm.2019.2944169 · Cited by 640
Self-citationTernGrad: Ternary Gradients to Reduce Communication in Distributed Deep Learning
Wen, W., Xu, C., Yan, F., Wu, C., Wang, Y., Chen, Y., & Li, H. H. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1705.07878 · Cited by 622
Reliable Federated Learning for Mobile Networks
Kang, J., Xiong, Z., Niyato, D., Zou, Y., Zhang, Y., & Guizani, M. (2020) · IEEE Wireless Communications · doi:10.1109/mwc.001.1900119 · Cited by 617
FedPAQ: A Communication-Efficient Federated Learning Method with Periodic Averaging and Quantization
Reisizadeh, A., Mokhtari, A., Hassani, H., Jadbabaie, A., & Pedarsani, R. (2019) · arXiv · doi:10.48550/arxiv.1909.13014 · Cited by 587 · cited 2×
“Additionally, we note that the communication load on the uplink channel induced by UVeQFed can be further reduced by allowing only part of the nodes to participate in each set of iterations [9,17].”
BackgroundSparse Communication for Distributed Gradient Descent
Aji, A. F., & Heafield, K. (2017) · Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d17-1045 · Cited by 585
signSGD: Compressed Optimisation for Non-Convex Problems
Bernstein, J., Wang, Y.-X., Azizzadenesheli, K., & Anandkumar, A. (2018) · arXiv · doi:10.48550/arxiv.1802.04434 · Cited by 541
The Effects of Adding Noise During Backpropagation Training on a Generalization Performance
An, G. (1996) · Neural Computation · doi:10.1162/neco.1996.8.3.643 · Cited by 475 · cited 1×
“This follows from the fact that the stochastic nature of the quantization error in UVeQFed results in its implementing a noisy variant of local SGD, which is known to be capable of boosting convergence and avoid local minimas when training deep neural networks with non-convex loss surfaces [57], as also observed in [4…”
BackgroundGray, R. M., & Stockham, T. G. (1993) · IEEE Transactions on Information Theory · doi:10.1109/18.256489 · Cited by 409
Zamir, R., & Feder, M. (1996) · IEEE Transactions on Information Theory · doi:10.1109/18.508838 · Cited by 375
Voronoi regions of lattices, second moments of polytopes, and quantization
Conway, J., & Sloane, N. (1982) · IEEE Transactions on Information Theory · doi:10.1109/tit.1982.1056483 · Cited by 321
Distributed Average Consensus With Dithered Quantization
Aysal, T. C., Coates, M. J., & Rabbat, M. G. (2008) · IEEE Transactions on Signal Processing · doi:10.1109/tsp.2008.927071 · Cited by 315
Quantization and Dither: A Theoretical Survey
Lipshitz, S. P., Wannamaker, R. A., & Vanderkooy, J. (1992) · Journal of the Audio Engineering Society · Cited by 286
Communication-efficient algorithms for statistical optimization
Zhang, Y., Duchi, J. C., & Wainwright, M. J. (2012) · 2012 IEEE 51st IEEE Conference on Decision and Control (CDC) · doi:10.1109/cdc.2012.6426691 · Cited by 283
Convergence of Update Aware Device Scheduling for Federated Learning at the Wireless Edge
Amiri, M. M., Gunduz, D., Kulkarni, S. R., & Poor, H. V. (2021) · IEEE Transactions on Wireless Communications · doi:10.1109/twc.2021.3052681 · Cited by 232
Self-citationOn universal quantization by randomized uniform/lattice quantizers
Zamir, R., & Feder, M. (1992) · IEEE Transactions on Information Theory · doi:10.1109/18.119699 · Cited by 221
Over-the-Air Federated Learning From Heterogeneous Data
Sery, T., Shlezinger, N., Cohen, K., & Eldar, Y. (2021) · IEEE Transactions on Signal Processing · doi:10.1109/tsp.2021.3090323 · Cited by 219
Self-citationCommunication-efficient algorithms for statistical optimization
(2013) · Journal of Machine Learning Research · doi:10.5555/2567709.2567769 · Cited by 195
Local SGD Converges Fast and Communicates Little
Stich, S. U. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1805.09767 · Cited by 191
Ziv, J. (1985) · IEEE Transactions on Information Theory · doi:10.1109/tit.1985.1057034 · Cited by 177
Lecture Notes on Information Theory
(2015) · Lecture Notes on Information Theory · doi:10.12720/lnit · Cited by 167
Natural Compression for Distributed Deep LearningPreprint
Horváth, S., Ho, C.-Y., Horváth, L., Sahu, A. N., Canini, M., & Richtárik, P. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1905.10988 · Cited by 156
A Unified Theory of Decentralized SGD with Changing Topology and Local Updates
Koloskova, A., Loizou, N., Boreiri, S., Jaggi, M., & Stich, S. U. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2003.10422 · Cited by 151
Multiterminal source coding with high resolution
Zamir, R., & Berger, T. (1999) · IEEE Transactions on Information Theory · doi:10.1109/18.746775 · Cited by 131
Federated Learning with Quantization Constraints
Shlezinger, N., Chen, M., Eldar, Y. C., Poor, H. V., & Cui, S. (2020) · ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp40776.2020.9054168 · Cited by 115
Self-citationHardware-Limited Task-Based Quantization
Shlezinger, N., Eldar, Y. C., & Rodrigues, M. R. D. (2019) · IEEE Transactions on Signal Processing · doi:10.1109/tsp.2019.2935864 · Cited by 110
Self-citationGenerating random vectors uniformly distributed inside and on the surface of different regions
Rubinstein, R. Y. (1982) · European Journal of Operational Research · doi:10.1016/0377-2217(82)90161-8 · Cited by 96
Stochastic Distributed Learning with Gradient Quantization and Variance Reduction
Horváth, S., Kovalev, D. A., Mishchenko, K., Stich, S. U., & Richtárik, P. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1904.05115 · Cited by 75
A vector quantization approach to universal noiseless coding and quantization
Chou, P. A., Effros, M., & Gray, R. M. (1996) · IEEE Transactions on Information Theory · doi:10.1109/18.508836 · Cited by 73
Distributed deep learning on edge-devices: Feasibility via adaptive compression
Hardy, C., Le Merrer, E., & Sericola, B. (2017) · 2017 IEEE 16th International Symposium on Network Computing and Applications (NCA) · doi:10.1109/nca.2017.8171350 · Cited by 73
Asymptotic Task-Based Quantization With Application to Massive MIMO
Shlezinger, N., Eldar, Y. C., & Rodrigues, M. R. D. (2019) · IEEE Transactions on Signal Processing · doi:10.1109/tsp.2019.2923149 · Cited by 69
Self-citationUpdate Aware Device Scheduling for Federated Learning at the Wireless Edge
Amiri, M. M., Gunduz, D., Kulkarni, S. R., & Vincent Poor, H. (2020) · 2020 IEEE International Symposium on Information Theory (ISIT) · doi:10.1109/isit44484.2020.9173960 · Cited by 67
Self-citationThe Communication-Aware Clustered Federated Learning Problem
Shlezinger, N., Rini, S., & Eldar, Y. C. (2020) · IEEE International Symposium on Information Theory (ISIT) · doi:10.1109/isit44484.2020.9174245 · Cited by 53
Self-citationOptimization of lattices for quantization
Agrell, E., & Eriksson, T. (1998) · IEEE Transactions on Information Theory · doi:10.1109/18.705561 · Cited by 49
Hardware-Limited Task-Based Quantization
Shlezinger, N., Eldar, Y. C., & Rodrigues, M. R. D. (2019) · IEEE International Workshop on Signal Processing Advances in Wireless Communications (SPAWC) · doi:10.1109/spawc.2019.8815422 · Cited by 44
Self-citationTask-Based Quantization for Recovering Quadratic Functions Using Principal Inertia Components
Salamatian, S., Shlezinger, N., Eldar, Y. C., & Médard, M. (2019) · IEEE International Symposium on Information Theory (ISIT) · doi:10.1109/isit.2019.8849346 · Cited by 44
Self-citationResults on lattice vector quantization with dithering
Kirac, A., & Vaidyanathan, P. P. (1996) · IEEE Transactions on Circuits and Systems II Analog and Digital Signal Processing · doi:10.1109/82.553397 · Cited by 34
On Tcebycheff's type inequalities
Ferentios, K. (1982) · Trabajos de Estadistica y de Investigacion Operativa · doi:10.1007/bf02888707 · Cited by 33
Task-based quantization with application to MIMO receivers
Shlezinger, N., & Eldar, Y. C. (2020) · Communications in Information and Systems · doi:10.4310/cis.2020.v20.n2.a3 · Cited by 33
Self-citationOn Tchebycheff's type inequalities
Ferentinos, K. (1982) · Hispana · Cited by 5
(2020) · International Conference on Learning Representations
Research works printed but not linked: 1
- 2020, Deep learning tutorial series,” MATLAB central file exchange, MathWorks Deep Learning Toolbox Team
Cited by
A Novel Framework for the Analysis and Design of Heterogeneous Federated Learning
Wang, J., Liu, Q., Liang, H., Joshi, G., & Poor, H. V. (2021) · IEEE Transactions on Signal Processing · doi:10.1109/tsp.2021.3106104 · Cited by 1,202
Self-citationFederated Learning for Internet of Things: Recent Advances, Taxonomy, and Open Challenges
Khan, L. U., Saad, W., Han, Z., Hossain, E., & Hong, C. S. (2021) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2021.3090430 · Cited by 892
Edge artificial intelligence for 6g: vision, enabling technologies, and applications
Letaief, K. B., Shi, Y., Lu, J., & Lu, J. (2022) · IEEE Journal on Selected Areas in Communications · doi:10.1109/jsac.2021.3126076 · Cited by 829
“Scalar quantization is a typical way to implement lossy compression for the high-dimensional gradient vectors by quantizing each of their entries to a finite-bit low precision value [123], [124], [125], which was further improved by the recent proposal of vector quantization [126], [127].”
BackgroundFederated learning meets blockchain in edge computing: Opportunities and challenges
Nguyen, D. C., Ding, M., Pham, Q.-V., Pathirana, P. N., Le, L. B., Seneviratne, A., Li, J., Niyato, D., & Poor, H. V. (2021) · IEEE Internet of Things Journal · doi:10.1109/jiot.2021.3072611 · Cited by 661
Self-citationDistributed Learning in Wireless Networks: Recent Progress and Future Challenges
Chen, M., Gunduz, D., Huang, K., Saad, W., Bennis, M., Feljan, A. V., & Poor, H. V. (2021) · IEEE Journal on Selected Areas in Communications · doi:10.1109/jsac.2021.3118346 · Cited by 548
Self-citation
Metadata
- Authors
- Shlezinger, N., Chen, M., Eldar, Y. C., Poor, H. V., & Cui, S.
- Affiliations
- Ben-Gurion University of the Negev (IL)
- Princeton University (US)
- Weizmann Institute of Science (IL)
- Shenzhen Research Institute of Big Data (CN)
- Venue
- IEEE Transactions on Signal Processing
- Published
- 2021
- Volume
- 69
- Pages
- 500-514
- ISSN
- 1053-587X (ISSN-L)
- 1941-0476
- OpenAlex
- W3033664100
- Cited by
- 275
- References
- 65
- Citations by year
- 2026: 26 · 2025: 45 · 2024: 58 · 2023: 60 · 2022: 48 · 2021: 35
- FWCI
- 14.35
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- IEEE Transactions on Signal Processing (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- Israel Science Foundation (0100101)
- Horizon 2020 Framework Programme (646804-ERC-COG-BNYQ)
- National Natural Science Foundation of China (NSFC-61629101)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Privacy-Preserving Technologies in Data
- Physical Sciences › Computer Science › Computer Networks and Communications › Distributed Sensor Networks and Detection Algorithms
- Physical Sciences › Computer Science › Artificial Intelligence › Stochastic Gradient Optimization Techniques
- Keywords
- federated learning, model compression, uplink channel, quantization distortion, decentralized learning, FedAvg, convergence analysis, privacy-preserving machine learning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.