Tackling System and Statistical Heterogeneity for Federated Learning with Adaptive Client Sampling
Open accessAbstract
Federated learning (FL) algorithms usually sample a fraction of clients in each round (partial participation) when the number of participants is large and the server’s communication bandwidth is limited. Recent works on the convergence analysis of FL have focused on unbiased client sampling, e.g., sampling uniformly at random, which suffers from slow wall-clock time for convergence due to high degrees of system heterogeneity and statistical heterogeneity. This paper aims to design an adaptive client sampling algorithm that tackles both system and statistical heterogeneity to minimize the wall-clock convergence time. We obtain a new tractable convergence bound for FL algorithms with arbitrary client sampling probabilities. Based on the bound, we analytically establish the relationship between the total learning time and sampling probabilities, which results in a non-convex optimization problem for training time minimization. We design an efficient algorithm for learning the unknown parameters in the convergence bound and develop a low-complexity algorithm to approximately solve the non-convex problem. Experimental results from both hardware prototype and simulation demonstrate that our proposed sampling scheme significantly reduces the convergence time compared to several baseline sampling schemes. Notably, our scheme in hardware prototype spends 73% less time than the uniform sampling baseline for reaching the same target loss.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
When time-division multiple-access (TDMA) protocols are considered, clients selected and scheduled earlier for model uploading could suffer from substantially shorter local model training time in a communication round and produce premature local models.
“When time-division multiple-access (TDMA) protocols are considered for clients to upload their local models, clients selected and scheduled earlier for model uploading could suffer from substantially shorter local model training time in a communication round and produce premature local models [179].”
From Data and model poisoning backdoor attacks on wireless federated learning, and the defense mechanisms: A comprehensive survey (2024) · cites this paper as [179]
Cites
Gradient-based learning applied to document recognition
Lecun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998) · Proceedings of the IEEE · doi:10.1109/5.726791 · Cited by 58,805 · see citing works
Boyd, S., & Vandenberghe, L. (2004) · doi:10.1017/cbo9780511804441 · Cited by 56,601 · see citing works
Federated Machine Learning: Concept and Applications
Yang, Q., Liu, Y., Chen, T., & Tong, Y. (2019) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3298981 · Cited by 6,322 · see citing works
Federated learning: Challenges, methods, and future directions
Li, T., Sahu, A. K., Talwalkar, A., & Smith, V. (2020) · IEEE Signal Processing Magazine · doi:10.1109/msp.2020.2975749 · Cited by 6,210 · see citing works
Federated Learning: Strategies for Improving Communication Efficiency
Konečný, J., McMahan, H. B., Yu, F. X., Richtárik, P., Suresh, A. T., & Bacon, D. (2016) · arXiv · doi:10.48550/arxiv.1610.05492 · Cited by 3,766 · see citing works
Adaptive Federated Learning in Resource Constrained Edge Computing Systems
Wang, S., Tuor, T., Salonidis, T., Leung, K. K., Makaya, C., He, T., & Chan, K. (2019) · IEEE Journal on Selected Areas in Communications · doi:10.1109/jsac.2019.2904348 · Cited by 2,232 · see citing works
Self-citationClient Selection for Federated Learning with Heterogeneous Resources in Mobile Edge
Nishio, T., & Yonetani, R. (2019) · ICC 2019 - 2019 IEEE International Conference on Communications (ICC) · doi:10.1109/icc.2019.8761315 · Cited by 1,875 · see citing works
On the Convergence of FedAvg on Non-IID Data
Li, X., Huang, K., Yang, W., Wang, S., & Zhang, Z. (2019) · arXiv · doi:10.48550/arxiv.1907.02189 · Cited by 1,575 · see citing works
Self-citationOptimizing Federated Learning on Non-IID Data with Reinforcement Learning
Wang, H., Kaplan, Z., Niu, D., & Li, B. (2020) · IEEE Conference on Computer Communications (INFOCOM) · doi:10.1109/infocom41043.2020.9155494 · Cited by 1,054 · see citing works
Federated Learning over Wireless Networks: Optimization Model Design and Analysis
Tran, N. H., Bao, W., Zomaya, A. Y., Nguyen, M. N. H., & Hong, C. (2019) · IEEE Conference on Computer Communications · doi:10.1109/infocom.2019.8737464 · Cited by 943 · see citing works
Yu, H., Yang, S., & Zhu, S.-H. (2018) · AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33015693 · Cited by 696 · see citing works
Scheduling Policies for Federated Learning in Wireless Networks
Yang, H. H., Liu, Z., Quek, T. Q. S., & Poor, H. V. (2020) · IEEE Transactions on Communications · doi:10.1109/tcomm.2019.2944169 · Cited by 640 · see citing works
Stochastic gradient descent, weighted sampling, and the randomized Kaczmarz algorithm
Needell, D., Srebro, N., & Ward, R. (2015) · Mathematical Programming · doi:10.1007/s10107-015-0864-7 · Cited by 431 · see citing works
Convergence Time Optimization for Federated Learning Over Wireless Networks
Chen, M., Poor, H. V., Saad, W., & Cui, S. (2021) · IEEE Transactions on Wireless Communications · doi:10.1109/twc.2020.3042530 · Cited by 403 · see citing works
Joint Device Scheduling and Resource Allocation for Latency Constrained Wireless Federated Learning
Shi, W., Zhou, S., Niu, Z., Jiang, M., & Geng, L. (2021) · IEEE Transactions on Wireless Communications · doi:10.1109/twc.2020.3025446 · Cited by 381 · see citing works
TiFL: A Tier-based Federated Learning System
Chai, Z., Ali, A. Z. E., Zawad, S., Truex, S., Anwar, A., Baracaldo, N., Zhou, Y., Ludwig, H., Yan, F., & Cheng, Y. (2020) · ACM International Symposium on High-Performance Parallel and Distributed Computing (HPDC) · doi:10.1145/3369583.3392686 · Cited by 352 · see citing works
Fast-Convergent Federated Learning
Nguyen, H. T., Sehwag, V., Hosseinalipour, S., Brinton, C. G., Chiang, M., & Poor, H. V. (2020) · IEEE Journal on Selected Areas in Communications · doi:10.1109/jsac.2020.3036952 · Cited by 281 · see citing works
Yu, H., Jin, R., & Yang, S. (2019) · arXiv · doi:10.48550/arxiv.1905.03817 · Cited by 244 · see citing works
Cost-Effective Federated Learning Design
Luo, B., Li, X., Wang, S., Huang, J., & Tassiulas, L. (2021) · IEEE Conference on Computer Communications (INFOCOM) · doi:10.1109/infocom42981.2021.9488679 · Cited by 236 · see citing works
Self-citationSCAFFOLD: Stochastic Controlled Averaging for On-Device Federated Learning.
Karimireddy, S. P., Kale, S., Mohri, M., Reddi, S. J., Stich, S. U., & Suresh, A. T. (2019) · Cited by 227 · see citing works
Stochastic Optimization with Importance Sampling for Regularized Loss Minimization
Zhao, P., & Zhang, T. (2015) · Rare & Special e-Zone (The Hong Kong University of Science and Technology) · Cited by 197 · see citing works
On the Convergence of Local Descent Methods in Federated LearningPreprint
Haddadpour, F., & Mahdavi, M. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1910.14425 · Cited by 176 · see citing works
Reisizadeh, A., Tziotis, I., Hassani, H., Mokhtari, A., & Pedarsani, R. (2022) · IEEE Journal on Selected Areas in Information Theory · doi:10.1109/jsait.2022.3205475 · Cited by 137 · see citing works
Device Sampling for Heterogeneous Federated Learning: Theory, Algorithms, and Implementation
Wang, S., Lee, M., Hosseinalipour, S., Morabito, R., Chiang, M., & Brinton, C. G. (2021) · IEEE Conference on Computer Communications (INFOCOM) · doi:10.1109/infocom42981.2021.9488906 · Cited by 123 · see citing works
Self-citationPain-FL: Personalized Privacy-Preserving Incentive for Federated Learning
Sun, P., Che, H., Wang, Z., Wang, Y., Wang, T., Wu, L., & Shao, H. (2021) · IEEE Journal on Selected Areas in Communications · doi:10.1109/jsac.2021.3118354 · Cited by 120 · see citing works
Cost-Effective Federated Learning in Mobile Edge Networks
Luo, B., Li, X., Wang, S., Huang, J.-W., & Tassiulas, L. (2021) · IEEE Journal on Selected Areas in Communications · doi:10.1109/jsac.2021.3118436 · Cited by 98 · see citing works
Self-citationOptimal Client Sampling for Federated LearningPreprint
Chen, W., Horváth, S., & Richtárik, P. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.13723 · Cited by 92 · see citing works
Network-Aware Optimization of Distributed Learning for Fog Computing
Tu, Y., Ruan, Y., Wagle, S., Brinton, C. G., & Joe-Wong, C. (2020) · IEEE INFOCOM 2020 - IEEE Conference on Computer Communications · doi:10.1109/infocom41043.2020.9155372 · Cited by 68 · see citing works
Resource-Efficient and Convergence-Preserving Online Participant Selection in Federated Learning
Jin, Y., Jiao, L., Qian, Z., Zhang, S., Lu, S., & Wang, X. (2020) · IEEE International Conference on Distributed Computing Systems (ICDCS) · doi:10.1109/icdcs47774.2020.00049 · Cited by 68 · see citing works
Faithful Edge Federated Learning: Scalability and Privacy
Zhang, M., Wei, E.-M., & Berry, R. (2021) · IEEE Journal on Selected Areas in Communications · doi:10.1109/jsac.2021.3118423 · Cited by 62 · see citing works
BLENDER: Enabling Local Search with a Hybrid Differential Privacy ModelPreprint
Avent, B., Korolova, A., Zeber, D., Hovden, T., & Livshits, B. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1705.00831 · Cited by 49 · see citing works
Adaptive sampling for SGD by exploiting side information
Gopal, S. (2016) · International Conference on Machine Learning · Cited by 43 · see citing works
Reisizadeh, A., Tziotis, I., Hassani, H., et al. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2012.14453 · Cited by 14 · see citing works
Federated Learning under Importance SamplingPreprint
Rizk, E., Vlaski, S., & Sayed, A. H. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2012.07383 · Cited by 3 · see citing works
OrbitalBrain: A Distributed Framework for Training ML Models in Space
Chabra, O., Li, C., Hsieh, K., et al. (2026) · NINeS · doi:10.4230/oasics.nines.2026.5 · Cited by 1 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Wan, Y., Qu, Y., Ni, W., Xiang, Y., Gao, L., & Hossain, E. (2024) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2024.3361451 · Cited by 152 · see citing works
“When time-division multiple-access (TDMA) protocols are considered for clients to upload their local models, clients selected and scheduled earlier for model uploading could suffer from substantially shorter local model training time in a communication round and produce premature local models [179].”
BackgroundOn Model Transmission Strategies in Federated Learning With Lossy Communications
Su, X., Zhou, Y., Cui, L., & Liu, J. (2023) · IEEE Transactions on Parallel and Distributed Systems · doi:10.1109/tpds.2023.3240883 · Cited by 47 · see citing works
- Authors
- Luo, B., Xiao, W., Wang, S., Huang, J., & Tassiulas, L.
- Affiliations
- Yale University (US)
- Chinese University of Hong Kong, Shenzhen (CN)
- IBM (United States) (US)
- Venue
- IEEE INFOCOM 2022 - IEEE Conference on Computer Communications
- Publisher
- IEEE
- Published
- 2022-05-02
- Pages
- 1739-1748
- OpenAlex
- W4226183928
- Cited by
- 229
- References
- 35
- Citations by year
- 2026: 23 · 2025: 74 · 2024: 68 · 2023: 55 · 2022: 9
- FWCI
- 28.46
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- IEEE Conference on Computer Communications (INFOCOM) (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- University of Hong Kong
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Privacy-Preserving Technologies in Data
- Physical Sciences › Computer Science › Artificial Intelligence › Stochastic Gradient Optimization Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Keywords
- federated learning, statistical heterogeneity, system heterogeneity, convergence analysis, convergence bounds, training time reduction, non-convex optimization, partial participation, communication bandwidth
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
