The DOI matches its Crossref record; 49 of the 49 references it cites were found in a registry or an index.
An Efficient Framework for Clustered Federated Learning
Open accessAbstract
We address the problem of federated learning (FL) where users are distributed and partitioned into clusters. This setup captures settings where different groups of users have their own objectives (learning tasks) but by aggregating their data with others in the same cluster (same learning task), they can leverage the strength in numbers in order to perform more efficient federated learning. For this new framework of clustered federated learning, we propose the Iterative Federated Clustering Algorithm (IFCA), which alternately estimates the cluster identities of the users and optimizes model parameters for the user clusters via gradient descent. We analyze the convergence rate of this algorithm first in a linear model with squared loss and then for generic strongly convex and smooth loss functions. We show that in both settings, with good initialization, IFCA is guaranteed to converge, and discuss the optimality of the statistical error rate. In particular, for the linear model with two clusters, we can guarantee that our algorithm converges as long as the initialization is slightly better than random. When the clustering structure is ambiguous, we propose to train the models by combining IFCA with the weight sharing technique in multi-task learning. In the experiments, we show that our algorithm can succeed even if we relax the requirements on initialization with random initialization and multiple restarts. We also present experimental results showing that our algorithm is efficient in non-convex problems such as neural networks. We demonstrate the benefits of IFCA over the baselines on several clustered FL benchmarks.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Gradient-based learning applied to document recognition
Lecun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998) · Proceedings of the IEEE · doi:10.1109/5.726791 · Cited by 58,805
Learning Multiple Layers of Features from Tiny Images
Krizhevsky, A. (2024) · doi:10.57702/zp44cu3g · Cited by 23,395
Least squares quantization in PCM
Lloyd, S. (1982) · IEEE Transactions on Information Theory · doi:10.1109/tit.1982.1056489 · Cited by 15,836
Communication-Efficient Learning of Deep Networks from Decentralized Data
McMahan, H. B., Moore, E., Ramage, D., Hampson, S., & Arcas, B. A. Y. (2016) · arXiv · doi:10.48550/arxiv.1602.05629 · Cited by 14,215
Overcoming catastrophic forgetting in neural networks
Kirkpatrick, J., Pascanu, R., Rabinowitz, N., Veness, J., Desjardins, G., Rusu, A. A., Milan, K., Quan, J., Ramalho, T., Grabska-Barwinska, A., et al. (2017) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1611835114 · Cited by 7,899
Caruana, R. (1997) · Machine Learning · doi:10.1023/a:1007379606734 · Cited by 6,636
Phase retrieval algorithms: a comparison
Fienup, J. R. (1982) · Applied Optics · doi:10.1364/ao.21.002758 · Cited by 5,596
Federated Optimization in Heterogeneous Networks
Li, T., Sahu, A. K., Zaheer, M., Sanjabi, M., Talwalkar, A., & Smith, V. (2018) · arXiv · doi:10.48550/arxiv.1812.06127 · Cited by 4,023 · cited 1×
“personal devices are usually non-i.i.d. Several formulations and solutions have been proposed to tackle this problem. A line of research focuses on learning a single global model from non-i.i.d. data [48, 33, 20, 36, 23, 29]. Other lines of research focus more on learning personalized models [37, 35, 7]. In particular…”
MethodIntroduction to the non-asymptotic analysis of random matrices
Vershynin, R. (2012) · Compressed Sensing · doi:10.1017/cbo9780511794308.006 · Cited by 2,150
Federated Learning with Non-IID Data
Zhao, Y., Li, M., Lai, L., Suda, N., Civin, D., & Chandra, V. (2018) · arXiv · doi:10.48550/arxiv.1806.00582 · Cited by 2,077
Robust and Communication-Efficient Federated Learning From Non-i.i.d. Data
Sattler, F., Wiedemann, S., Muller, K.-R., & Samek, W. (2020) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2019.2944481 · Cited by 1,725
On the Convergence of FedAvg on Non-IID Data
Li, X., Huang, K., Yang, W., Wang, S., & Zhang, Z. (2019) · arXiv · doi:10.48550/arxiv.1907.02189 · Cited by 1,575
Federated Optimization: Distributed Machine Learning for On-Device Intelligence
Konečný, J., McMahan, H. B., Ramage, D., & Richtárik, P. (2016) · arXiv · doi:10.48550/arxiv.1610.02527 · Cited by 1,563
Gradient Episodic Memory for Continual Learning
Lopez-Paz, D., & Ranzato, M. (2017) · arXiv · doi:10.48550/arxiv.1706.08840 · Cited by 1,403
Sattler, F., Muller, K.-R., & Samek, W. (2021) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2020.3015958 · Cited by 1,355
Federated Learning of Deep Networks using Model AveragingPreprint
McMahan, H. B., Moore, E., Ramage, D., & y Arcas, B. A. (2016) · arXiv (Cornell University) · Cited by 1,288
Hogwild: A Lock-Free Approach to Parallelizing Stochastic Gradient Descent
Niu, F., Recht, B., Re, C., & Wright, S. J. (2011) · arXiv · doi:10.48550/arxiv.1106.5730 · Cited by 1,211 · cited 1×
“on guarantees. We discuss other related work in the following. Federated Learning and non-i.i.d. data: Learning with a distributed computing framework has been studied extensively in various settings [49, 31, 21]. As mentioned in Section 1, Federated Learning [27, 26, 16, 12] is one of the modern distributed learning…”
BackgroundFederated Learning for Mobile Keyboard Prediction
Hard, A., Rao, K., Mathews, R., Ramaswamy, S., Beaufays, F., Augenstein, S., Eichner, H., Kiddon, C., & Ramage, D. (2018) · arXiv · doi:10.48550/arxiv.1811.03604 · Cited by 1,148 · cited 1×
“As mentioned in Section I, federated learning [1]–[3], [17] is one of the modern distributed learning frameworks that aims to better utilize the data and computing power on edge devices.”
BackgroundParallelized Stochastic Gradient Descent
Zinkevich, M., Weimer, M., Li, L., & Smola, A. J. (2010) · Cited by 1,124
An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networks
Goodfellow, I. J., Mirza, M., Xiao, D., Courville, A., & Bengio, Y. (2013) · arXiv · doi:10.48550/arxiv.1312.6211 · Cited by 1,089
Scaling distributed machine learning with the parameter server
Li, M., Andersen, D. G., Park, J. W., Smola, A. J., Ahmed, A. A., Josifovski, V., Long, J. E., Shekita, E. J., & Su, B.-Y. (2014) · Operating Systems Design and Implementation · doi:10.5555/2685048.2685095 · Cited by 1,021
Scaling Distributed Machine Learning with the Parameter Server
Li, M. (2014) · Proceedings of the 2014 International Conference on Big Data Science and Computing · doi:10.1145/2640087.2644155 · Cited by 979
LEAF: A Benchmark for Federated Settings
Caldas, S., Duddu, S. M. K., Wu, P., Li, T., Konečný, J., McMahan, H. B., Smith, V., & Talwalkar, A. (2018) · arXiv · doi:10.48550/arxiv.1812.01097 · Cited by 894
High-Dimensional Statistics: A Non-Asymptotic Viewpoint
Wainwright, M. J. (2019) · Cambridge University Press eBooks · doi:10.1017/9781108627771 · Cited by 882
Effective Gaussian mixture learning for video background subtraction
Lee, D.-S. (2005) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2005.102 · Cited by 823
Low-rank matrix completion using alternating minimization
Jain, P., Netrapalli, P., & Sanghavi, S. (2013) · Proceedings of the forty-fifth annual ACM symposium on Theory of Computing · doi:10.1145/2488608.2488693 · Cited by 817
On Convergence Properties of the EM Algorithm for Gaussian Mixtures
Xu, L., & Jordan, M. I. (1996) · Neural Computation · doi:10.1162/neco.1996.8.1.129 · Cited by 756
Phase retrieval in crystallography and optics
Millane, R. P. (1990) · Journal of the Optical Society of America A · doi:10.1364/josaa.7.000394 · Cited by 720
Phase Retrieval Using Alternating Minimization
Netrapalli, P., Jain, P., & Sanghavi, S. (2015) · IEEE Transactions on Signal Processing · doi:10.1109/tsp.2015.2448516 · Cited by 535
A maximum likelihood methodology for clusterwise linear regression
DeSarbo, W. S., & Cron, W. L. (1988) · Journal of Classification · doi:10.1007/bf01897167 · Cited by 518
Optimal Distributed Online Prediction using Mini-Batches
Dekel, O., Gilad-Bachrach, R., Shamir, O., & Xiao, L. (2010) · arXiv (Cornell University) · doi:10.48550/arxiv.1012.1367 · Cited by 512
Hanson-Wright inequality and sub-gaussian concentration
Rudelson, M., & Vershynin, R. (2013) · Electronic Communications in Probability · doi:10.1214/ecp.v18-2865 · Cited by 502
Statistical guarantees for the EM algorithm: From population to sample-based analysis
Balakrishnan, S., Wainwright, M. J., & Yu, B. (2017) · The Annals of Statistics · doi:10.1214/16-aos1435 · Cited by 384
Improving Federated Learning Personalization via Model Agnostic Meta Learning
Jiang, Y., Konečný, J., Rush, K., & Kannan, S. (2019) · arXiv · doi:10.48550/arxiv.1909.12488 · Cited by 381 · cited 3×
“This problem has recently received much attention [4]–[6].”
BackgroundMohri, M., Sivek, G., & Suresh, A. T. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1902.00146 · Cited by 373
Li, L., Xu, W., Chen, T., Giannakis, G. B., & Ling, Q. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33011544 · Cited by 373
Personalized Federated Learning: A Meta-Learning Approach
Fallah, A., Mokhtari, A., & Ozdaglar, A. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2002.07948 · Cited by 345
Three Approaches for Personalization with Applications to Federated Learning
Mansour, Y., Mohri, M., Ro, J. H., & Suresh, A. T. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2002.10619 · Cited by 301
Robust Federated Learning in a Heterogeneous EnvironmentPreprint
Ghosh, A., Hong, J., Yin, D., & Ramchandran, K. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1906.06629 · Cited by 299
Self-citationFederated Meta-Learning with Fast Convergence and Efficient Communication
Chen, F., Luo, M., Dong, Z., Li, Z., & He, X. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1802.07876 · Cited by 254
On the Convergence of Federated Optimization in Heterogeneous Networks.Preprint
Sahu, A. K., Li, T., Sanjabi, M., Zaheer, M., Talwalkar, A., & Smith, V. (2018) · arXiv (Cornell University) · Cited by 237
Clustering approach for hybrid recommender system
Li, Q., & Kim, B. M. (2004) · IEEE/WIC/ACM International Conference on Web Intelligence · doi:10.1109/wi.2003.1241167 · Cited by 155
GIANT: Globally Improved Approximate Newton Method for Distributed OptimizationPreprint
Wang, S., Roosta, F., Xu, P., & Mahoney, M. W. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1709.03528 · Cited by 121
Phase Retrieval With Random Gaussian Sensing Vectors by Alternating Projections
Waldspurger, I. (2018) · IEEE Transactions on Information Theory · doi:10.1109/tit.2018.2800663 · Cited by 92
Alternating Minimization for Mixed Linear RegressionPreprint
Yi, X., Caramanis, C., & Sanghavi, S. (2013) · arXiv (Cornell University) · doi:10.48550/arxiv.1310.3745 · Cited by 85
Ten Steps of EM Suffice for Mixtures of Two GaussiansPreprint
Daskalakis, C., Tzamos, C., & Zampetakis, M. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1609.00368 · Cited by 38
Distributed Newton Can Communicate Less and Resist Byzantine WorkersPreprint
Ghosh, A., Maity, R. K., & Mazumdar, A. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.08737 · Cited by 37
Self-citationLEARNING MIXTURES OF LINEAR CLASSIFIERSPreprint
Sun, Y., Ioannidis, S., & Andrea Montanari, A. (2013) · arXiv (Cornell University) · doi:10.48550/arxiv.1311.2547 · Cited by 15
FedGroup: Accurate Federated Learning via Decomposed Similarity-Based ClusteringPreprint
Duan, M., Liu, D., Ji, X., Liu, R. P., Liang, L., Chen, X., & Tan, Y. (2020) · arXiv (Cornell University) · Cited by 6
Cited by
A survey on security and privacy of federated learning
Mothukuri, V., Parizi, R. M., Pouriyeh, S., Huang, Y., Dehghantanha, A., & Srivastava, G. (2021) · Future Generation Computer Systems · doi:10.1016/j.future.2020.10.007 · Cited by 1,480
A Survey on Federated Learning Systems: Vision, Hype and Reality for Data Privacy and Protection
Li, Q., Wen, Z., Wu, Z., Hu, S., Wang, N., Li, Y., Liu, X., & He, B. (2023) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2021.3124599 · Cited by 1,407
Federated learning on non-IID data: A survey
Zhu, H., Xu, J., Liu, S., & Jin, Y. (2021) · Neurocomputing · doi:10.1016/j.neucom.2021.07.098 · Cited by 1,062
FedProto: Federated Prototype Learning across Heterogeneous Clients
Tan, Y., Long, G., LIU, L., Zhou, T., Lu, Q., Jiang, J., & Zhang, C. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i8.20819 · Cited by 761
Distributed Learning in Wireless Networks: Recent Progress and Future Challenges
Chen, M., Gunduz, D., Huang, K., Saad, W., Bennis, M., Feljan, A. V., & Poor, H. V. (2021) · IEEE Journal on Selected Areas in Communications · doi:10.1109/jsac.2021.3118346 · Cited by 548
Metadata
- Authors
- Ghosh, A., Chung, J., Yin, D., & Ramchandran, K.
- Venue
- IEEE Transactions on Information Theory
- Published
- 2022-12
- Volume
- 68
- Issue
- 12
- Pages
- 8076-8091
- ISSN
- 0018-9448 (ISSN-L)
- 1557-9654
- OpenAlex
- W3034163621
- Cited by
- 623
- References
- 49
- FWCI
- 39.42
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.