Federated Learning: Challenges, Methods, and Future Directions
Open accessThe DOI matches its Crossref record; 87 of the 87 references it cites were found in a registry or an index.
Abstract
Federated learning involves training statistical models over remote devices or siloed data centers, such as mobile phones or hospitals, while keeping data localized. Training in heterogeneous and potentially massive networks introduces novel challenges that require a fundamental departure from standard approaches for large-scale machine learning, distributed optimization, and privacy-preserving data analysis. In this article, we discuss the unique characteristics and challenges of federated learning, provide a broad overview of current approaches, and outline several directions of future work that are relevant to a wide range of research communities.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Trusted Distributed AI extends the principles of federated and collaborative learning with trust guarantees, ensuring that distributed models can be trained and deployed without exposing sensitive data or compromising model integrity [235,236].”
From Anomaly detection and root-cause identification in microservices: a survey (2026) · cites this paper as [235]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The actual process for aggregating parameters depends on the network topology, as nodes might be segregated into sub-networks due to geographical or legal constraints (see Fig. 2).”
From The future of digital health with federated learning (2020) · cites this paper as [10]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“It is important to note that FL does not solve all potential privacy issues and—similar to ML algorithms in general—will always carry some risks.”
From The future of digital health with federated learning (2020) · cites this paper as [10]
Experimental — accuracy not yet measuredThe check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“very little behavioral work has been done on eliciting preferences for differential privacy [144, 6].”
From Advances and Open Problems in Federated Learning (2021) · cites this paper as [6]
Experimental — accuracy not yet measuredIn the citing paper
“FL is a paradigm that aims to collaborate on data management and privacy issues by using evaluation metrics without transferring samples [47–49].”
From A comprehensive survey on federated learning techniques for healthcare informatics (2023) · cites this paper as [48]
2 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Boyd, S., Parikh, N., Chu, E., Peleato, B., & Eckstein, J. (2011) · Foundations and Trends® in Machine Learning · doi:10.1561/2200000016 · Cited by 17,075
Communication-Efficient Learning of Deep Networks from Decentralized Data
McMahan, H. B., Moore, E., Ramage, D., Hampson, S., & Arcas, B. A. Y. (2016) · arXiv · doi:10.48550/arxiv.1602.05629 · Cited by 14,215 · cited 5×
“This has led to a growing interest in federated learning [31], which explores training statistical models directly on remote devices.”
BackgroundMethodInfluentialCalibrating Noise to Sensitivity in Private Data Analysis
Dwork, C., McSherry, F., Nissim, K., & Smith, A. (2006) · Lecture Notes in Computer Science · doi:10.1007/11681878_14 · Cited by 7,272
Federated Machine Learning: Concept and Applications
Yang, Q., Liu, Y., Chen, T., & Tong, Y. (2019) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3298981 · Cited by 6,322 · cited 1×
“Federated learning methods have been deployed by major service providers [11, 124], and play a critical role in supporting privacy-sensitive applications where the training data are distributed at the edge [e.g., 5, 46, 51, 89, 105, 127, 139].”
MethodFog computing and its role in the internet of things
Bonomi, F., Milito, R., Zhu, J., & Addepalli, S. (2012) · Proceedings of the first edition of the MCC workshop on Mobile cloud computing · doi:10.1145/2342509.2342513 · Cited by 6,244
Federated Optimization in Heterogeneous Networks
Li, T., Sahu, A. K., Zaheer, M., Sanjabi, M., Talwalkar, A., & Smith, V. (2018) · arXiv · doi:10.48550/arxiv.1812.06127 · Cited by 4,023 · cited 7×
“Indeed, when data is not identically distributed across devices in the network, methods such as FedAvg have been shown to diverge in practice [65, 75].”
BackgroundMethodInfluentialSelf-citationPractical Secure Aggregation for Privacy-Preserving Machine Learning
Bonawitz, K., Ivanov, V., Kreuter, B., Marcedone, A., McMahan, H. B., Patel, S., Ramage, D., Segal, A., & Seth, K. (2017) · Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security · doi:10.1145/3133956.3133982 · Cited by 3,956 · cited 1×
“quires that the updates are also private to the server. Current works that aim to improve the privacy of federated learning typically build upon previous classical cryptographic protocols such as SMC [10, 42] and differential privacy [3, 8, 41, 76]. Bonawitz et al. [10] introduce an SMC protocol to protect individual…”
BackgroundFederated Learning: Strategies for Improving Communication Efficiency
Konečný, J., McMahan, H. B., Yu, F. X., Richtárik, P., Suresh, A. T., & Bacon, D. (2016) · arXiv · doi:10.48550/arxiv.1610.05492 · Cited by 3,766 · cited 1×
“Nevertheless, several works have provided practical strategies in federated settings, such as forcing the updating models to be sparse and low-rank; performing quantization with structured random rotations [59]; using lossy compression and dropout to reduce server-to-device communication [15]; and applying Golomb loss…”
BackgroundTowards Federated Learning at Scale: System Design
Bonawitz, K., Eichner, H., Grieskamp, W., Huba, D., Ingerman, A., Ivanov, V., Kiddon, C., Konečný, J., Mazzocchi, S., McMahan, H. B., et al. (2019) · arXiv · doi:10.48550/arxiv.1902.01046 · Cited by 2,334 · cited 17×
“Indeed, when data is not identically distributed across devices in the network, methods such as FedAvg have been shown to diverge in practice [65, 75].”
BackgroundMethodInfluentialA Survey on Wearable Sensor-Based Systems for Health Monitoring and Prognosis
Pantelopoulos, A., & Bourbakis, N. G. (2010) · IEEE Transactions on Systems Man and Cybernetics Part C (Applications and Reviews) · doi:10.1109/tsmcc.2009.2032660 · Cited by 2,296 · cited 1×
“te nature of the data and the limited connectivity of each device. Federated learning methods can help to train models that efficiently adapt to changes in these systems while maintaining user privacy [84, 98]. 1.1 Problem Formulation The canonical federated learning problem involves learning a single, global statistic…”
BackgroundInfluentialFederated Learning with Non-IID Data
Zhao, Y., Li, M., Lai, L., Suda, N., Civin, D., & Chandra, V. (2018) · arXiv · doi:10.48550/arxiv.1806.00582 · Cited by 2,077 · cited 5×
“Zhao et al. [138] explore transfer learning for personalization by running FedAvg after training a global model centrally on some shared proxy data.”
BackgroundInfluentialTinyDB: an acquisitional query processing system for sensor networks
Madden, S. R., Franklin, M. J., Hellerstein, J. M., & Hong, W. (2005) · ACM Transactions on Database Systems · doi:10.1145/1061318.1061322 · Cited by 1,899 · cited 1×
“Indeed, computing simple queries across distributed, low-powered devices is a decades-long area of research that has been explored under the purview of query processing in sensor networks, computing at the edge, and fog computing [6], [30].”
BackgroundClient Selection for Federated Learning with Heterogeneous Resources in Mobile Edge
Nishio, T., & Yonetani, R. (2019) · ICC 2019 - 2019 IEEE International Conference on Communications (ICC) · doi:10.1109/icc.2019.8761315 · Cited by 1,875 · cited 2×
“For example, Nishio and Yonetani [36] explore novel devicesampling policies based on systems resources, with the aim being for the server to aggregate as many device updates as possible within a predefined time window.”
BackgroundClient Selection for Federated Learning with Heterogeneous Resources in Mobile Edge
Nishio, T., & Yonetani, R. (2018) · arXiv · doi:10.48550/arxiv.1804.08333 · Cited by 1,875 · cited 2×
“For example, Nishio and Yonetani [36] explore novel devicesampling policies based on systems resources, with the aim being for the server to aggregate as many device updates as possible within a predefined time window.”
BackgroundExploiting Unintended Feature Leakage in Collaborative Learning
Melis, L., Song, C., De Cristofaro, E., & Shmatikov, V. (2019) · 2019 IEEE Symposium on Security and Privacy (SP) · doi:10.1109/sp.2019.00029 · Cited by 1,734 · cited 1×
“the need to keep raw data on each device local in federated settings. However, sharing other information such as model updates as part of the training process can also leak sensitive user information [8, 17, 39, 78]. For instance, Carlini et al. [17] demonstrate that one can extract sensitive text patterns, e.g., a sp…”
BackgroundPrivacy-preserving data mining
Agrawal, R., & Srikant, R. (2000) · Proceedings of the 2000 ACM SIGMOD international conference on Management of data · doi:10.1145/342009.335438 · Cited by 1,719 · cited 1×
“Privacy-preserving learning has been extensively studied by the machine learning [e.g., 76], systems [e.g., 4, 11], and theory [e.g., 38, 69] communities.”
On the Convergence of FedAvg on Non-IID Data
Li, X., Huang, K., Yang, W., Wang, S., & Zhang, Z. (2019) · arXiv · doi:10.48550/arxiv.1907.02189 · Cited by 1,575 · cited 2×
“, convexity [27] or uniformly bounded gradients [52].”
BackgroundFederated Multi-Task LearningPreprint
Smith, V., Chiang, C.-K., Sanjabi, M., & Talwalkar, A. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1705.10467 · Cited by 1,320 · cited 4×
“Theoretically, although several recent works have investigated convergence guarantees of variants of federated learning methods [52], few analyses allow for low participation [25], [42] or study directly the effect of dropped devices [50].”
BackgroundMethodInfluentialSelf-citationHogwild: A Lock-Free Approach to Parallelizing Stochastic Gradient Descent
Niu, F., Recht, B., Re, C., & Wright, S. J. (2011) · arXiv · doi:10.48550/arxiv.1106.5730 · Cited by 1,211 · cited 1×
“usceptible to stragglers in the face of device variability. Asynchronous schemes are an attractive approach to mitigate stragglers in heterogeneous environments, particularly in shared-memory systems [27, 30, 48, 92, 141]. However, they typically rely on bounded-delay assumptions to control the degree of staleness, wh…”
BackgroundFederated Learning for Mobile Keyboard Prediction
Hard, A., Rao, K., Mathews, R., Ramaswamy, S., Beaufays, F., Augenstein, S., Eichner, H., Kiddon, C., & Ramage, D. (2018) · arXiv · doi:10.48550/arxiv.1811.03604 · Cited by 1,148 · cited 2×
“Smartphones By jointly learning user behavior across a large pool of mobile phones, statistical models can power applications such as next-word prediction [17].”
BackgroundByzantine-Robust Distributed Learning: Towards Optimal Statistical Rates
Yin, D., Chen, Y., Ramchandran, K., & Bartlett, P. (2018) · arXiv · doi:10.48550/arxiv.1803.01498 · Cited by 1,130 · cited 2×
“Theoretically, although several recent works have investigated convergence guarantees of variants of federated learning methods [52], few analyses allow for low participation [25], [42] or study directly the effect of dropped devices [50].”
BackgroundPrivacy Preserving Data Mining
Lindell, Y., & Pinkas, B. (2000) · Lecture Notes in Computer Science · doi:10.1007/3-540-44598-6_3 · Cited by 1,077
Differentially Private Federated Learning: A Client Level Perspective
Geyer, R. C., Klein, T., & Nabi, M. (2017) · arXiv · doi:10.48550/arxiv.1712.07557 · Cited by 1,052 · cited 3×
“Current works that aim to improve the privacy of federated learning typically build upon previous classical cryptographic protocols such as SMC [10, 42] and differential privacy [3, 8, 41, 76].”
BackgroundFederated learning of predictive models from federated Electronic Health Records
Brisimi, T. S., Chen, R., Mela, T., Olshevsky, A., Paschalidis, I. C., & Shi, W. (2018) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2018.01.007 · Cited by 997 · cited 1×
“Federated learning methods have been deployed in practice by major companies [5], [41] and play a critical role in supporting privacy-sensitive applications where training data are distributed at the edge [8], [19].”
MethodOn Early Stopping in Gradient Descent Learning
Yao, Y., Rosasco, L., & Caponnetto, A. (2007) · Constructive Approximation · doi:10.1007/s00365-006-0663-2 · Cited by 970 · cited 1×
“Indeed, it is well known that optimization methods used for machine learning can tolerate a lack of precision; this error can, in fact, help with generalization [49].”
BackgroundEdge-centric Computing: Vision and Challenges
Garcia Lopez, P., Montresor, A., Epema, D., Datta, A., Higashino, T., Iamnitchi, A., Barcellos, M., Felber, P., & Riviere, E. (2015) · ACM SIGCOMM Computer Communication Review · doi:10.1145/2831347.2831354 · Cited by 934 · cited 1×
“es across distributed, low-powered devices is a decades-long area of research that has been explored under the purview of query processing in sensor networks, computing at the edge, and fog computing [12, 29, 40, 49, 74]. Recent works have also considered training machine learning models centrally but serving and stor…”
BackgroundLEAF: A Benchmark for Federated Settings
Caldas, S., Duddu, S. M. K., Wu, P., Li, T., Konečný, J., McMahan, H. B., Smith, V., & Talwalkar, A. (2018) · arXiv · doi:10.48550/arxiv.1812.01097 · Cited by 894 · cited 1×
“It is critical for the broader research communities to further build upon existing implementations and benchmarking tools, such as LEAF [16] and TensorFlow Federated [1], to facilitate both the reproducibility of empirical results and the dissemination of new solutions for federated learning.”
Self-citationLearning Differentially Private Recurrent Language Models
McMahan, H. B., Ramage, D., Talwar, K., & Zhang, L. (2017) · arXiv · doi:10.48550/arxiv.1710.06963 · Cited by 849
Machine Learning Classification over Encrypted Data
Bost, R., Popa, R. A., Tu, S., & Goldwasser, S. (2015) · Proceedings 2015 Network and Distributed System Security Symposium · doi:10.14722/ndss.2015.23241 · Cited by 674 · cited 1×
“We refer interested readers to [7] for a more comprehensive review of the approaches based on homomorphic encryption and SMC.”
BackgroundThe Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networks
Carlini, N., Liu, C., Erlingsson, Ú., Kos, J., & Song, D. (2018) · arXiv · doi:10.48550/arxiv.1802.08232 · Cited by 629 · cited 1×
“[10] demonstrate that one can extract sensitive text patterns, e.”
BackgroundSheller, M. J., Reina, G. A., Edwards, B., Martin, J., & Bakas, S. (2019) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-11723-8_9 · Cited by 609
FedPAQ: A Communication-Efficient Federated Learning Method with Periodic Averaging and Quantization
Reisizadeh, A., Mokhtari, A., Hassani, H., Jadbabaie, A., & Pedarsani, R. (2019) · arXiv · doi:10.48550/arxiv.1909.13014 · Cited by 587
Deep learning with Elastic Averaging SGDPreprint
Zhang, S., Choromanska, A., & LeCun, Y. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1412.6651 · Cited by 546 · cited 4×
“, workers running independently without synchronization) schemes are both commonly used to parallelize iterative optimization algorithms, with each approach having advantages and disadvantages [37], [53].”
BackgroundInfluentialSplit learning for health: Distributed deep learning without sharing raw patient data
Vepakomma, P., Gupta, O., Swedish, T., & Raskar, R. (2018) · arXiv · doi:10.48550/arxiv.1812.00564 · Cited by 513
Yu, H., Yang, S., & Zhu, S. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33015693 · Cited by 497
Semi-supervised Knowledge Transfer for Deep Learning from Private Training Data
Papernot, N., Abadi, M., Erlingsson, Ú., Goodfellow, I., & Talwar, K. (2016) · arXiv · doi:10.48550/arxiv.1610.05755 · Cited by 492 · cited 1×
“Such sample-level privacy can be achieved in many learning tasks [2, 7, 22, 53, 85, 86].”
BackgroundMobile fog: a programming model for large-scale applications on the internet of things
Hong, K., Lillethun, D., Ramachandran, U., Ottenwälder, B., & Koldehofe, B. (2013) · Proceedings of the second ACM SIGCOMM workshop on Mobile cloud computing · doi:10.1145/2491266.2491270 · Cited by 486 · cited 1×
“es across distributed, low-powered devices is a decades-long area of research that has been explored under the purview of query processing in sensor networks, computing at the edge, and fog computing [12, 29, 40, 49, 74]. Recent works have also considered training machine learning models centrally but serving and stor…”
BackgroundFair Resource Allocation in Federated Learning
Li, T., Sanjabi, M., Beirami, A., & Smith, V. (2019) · arXiv · doi:10.48550/arxiv.1905.10497 · Cited by 475 · cited 2×
“Recent works have proposed modified modeling approaches that aim to reduce the variance of the model performance across devices [19], [26], [33].”
BackgroundSelf-citationJeong, E., Oh, S., Kim, H., Park, J., Bennis, M., & Kim, S.-L. (2018) · arXiv · doi:10.48550/arxiv.1811.11479 · Cited by 465
Privacy-Preserving Ridge Regression on Hundreds of Millions of Records
Nikolaenko, V., Weinsberg, U., Ioannidis, S., Joye, M., Boneh, D., & Taft, N. (2013) · 2013 IEEE Symposium on Security and Privacy · doi:10.1109/sp.2013.30 · Cited by 439 · cited 1×
“Beyond differential privacy, homomorphic encryption can be used to secure the learning process by computing on encrypted data, although it has currently been applied in limited settings, e.g., training linear models [82] or involving only a few entities [133].”
MethodCommunication-Efficient Distributed Optimization using an Approximate Newton-type Method
Shamir, O., Srebro, N., & Zhang, T. (2013) · arXiv · doi:10.48550/arxiv.1312.7853 · Cited by 424 · cited 1×
“setting [68, 93, 104, 108, 120, 121, 122, 125, 136, 140].”
Adaptive Gradient-Based Meta-Learning MethodsPreprint
Khodak, M., Balcan, M.-F., & Talwalkar, A. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1906.02717 · Cited by 402 · cited 2×
“Modeling heterogeneous data There exists a large body of literature in machine learning that models statistical heterogeneity via methods such as metalearning and multitask learning; these ideas have been recently extended to the federated setting [12], [14], [21].”
BackgroundSelf-citationLian, X., Zhang, C., Zhang, H., Hsieh, C., Zhang, W., & Liu, J. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1705.09056 · Cited by 395 · cited 1×
“For instance, numerous methods have been proposed to tackle expensive communication in the optimization and signal processing communities [28], [40], [43].”
MethodWang, J., & Joshi, G. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1808.07576 · Cited by 381
Mohri, M., Sivek, G., & Suresh, A. T. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1902.00146 · Cited by 373 · cited 2×
“Recent works have proposed modified modeling approaches that aim to reduce the variance of the model performance across devices [19], [26], [33].”
BackgroundMethodProtecting Privacy Using k-Anonymity
El Emam, K., & Dankar, F. K. (2008) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m2716 · Cited by 333 · cited 1×
“Despite the fact that differential privacy is the de facto metric for privacy in machine learning, there are many other privacy definitions, such as k-anonymity [15] and d-presence [34], which may be applicable to different learning problems.”
BackgroundOne-Shot Federated LearningPreprint
Guha, N., Talwalkar, A., & Smith, V. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1902.11175 · Cited by 325 · cited 1×
“Similarly, one-shot/few-shot heuristics [44, 45, 134] have recently been proposed for the federated setting, but have yet to be theoretically analyzed or evaluated at scale.”
BackgroundSelf-citationExpanding the Reach of Federated Learning by Reducing Client Resource Requirements
Caldas, S., Konečny, J., McMahan, H. B., & Talwalkar, A. (2018) · arXiv · doi:10.48550/arxiv.1812.07210 · Cited by 312 · cited 1×
“54 IEEE SIGNAL PROCESSING MAGAZINE | May 2020 | structured random rotations [22], and using lossy compression and dropout to reduce server-to-device communication [9].”
Self-citationFederated Meta-Learning with Fast Convergence and Efficient Communication
Chen, F., Luo, M., Dong, Z., Li, Z., & He, X. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1802.07876 · Cited by 254
Adaptive Neural Networks for Efficient Inference
Bolukbasi, T., Wang, J., Dekel, O., & Saligrama, V. (2017) · arXiv · doi:10.48550/arxiv.1702.07811 · Cited by 251 · cited 1×
“Similar comprehensive analyses have been performed for efficient neural network inference [3] and are necessary to compare communication-reduction techniques for federated learning in a meaningful way.”
BackgroundDifferentially Private Learning with Adaptive ClippingPreprint
Andrew, G., Thakkar, O. D., McMahan, H. B., & Ramaswamy, S. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1905.03871 · Cited by 250 · cited 2×
“These approaches have a number of hyperparameters that affect communication and accuracy that must be carefully chosen, though follow up work [113] proposes adaptive gradient clipping strategies to help alleviate this issue.”
BackgroundProtection Against Reconstruction and Its Applications in Private Federated Learning
Bhowmick, A., Duchi, J. C., Freudiger, J., Kapoor, G., & Rogers, R. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1812.00984 · Cited by 240 · cited 4×
“Current works that aim to improve the privacy of federated learning typically build upon previous classical cryptographic protocols such as SMC [10, 42] and differential privacy [3, 8, 41, 76].”
BackgroundInfluentialFederated Learning for Emoji Prediction in a Mobile Keyboard
Ramaswamy, S., Mathews, R., Rao, K., & Beaufays, F. (2019) · arXiv · doi:10.48550/arxiv.1906.04329 · Cited by 231 · cited 1×
“By jointly learning user behavior across a large pool of mobile phones, statistical models can power applications such as next-word prediction, face detection, and voice recognition [46, 89].”
BackgroundVariational Federated Multi-Task LearningPreprint
Corinzia, L., Beuret, A., & Buhmann, J. M. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1906.06268 · Cited by 230 · cited 2×
“Another approach [26] models the star topology as a Bayesian network and performs variational inference during learning.”
BackgroundCommunication Compression for Decentralized Training
Tang, H., Gan, S., Zhang, C., Zhang, T., & Liu, J. (2018) · arXiv · doi:10.48550/arxiv.1803.06443 · Cited by 206 · cited 1×
“, [45]), the assumptions made do not take into consideration common characteristics of the federated setting, such as low device participation or locally updating optimization methods.”
BackgroundFederated Collaborative Filtering for Privacy-Preserving Personalized Recommendation System
Ammad-ud-din, M., Ivannikova, E., Khan, S. A., Oyomno, W., Fu, Q., Tan, K. E., & Flanagan, A. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1901.09888 · Cited by 203
Don't Use Large Mini-Batches, Use Local SGD
Lin, T., Stich, S. U., Patel, K. K., & Jaggi, M. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1808.07217 · Cited by 193 · cited 2×
“Finally, hierarchical communication patterns have also been proposed [68, 70] to further ease the burden on the central server, by first leveraging edge servers to aggregate the updates from edge devices and then relying on a cloud server to aggregate updates from edge servers.”
BackgroundCoCoA: A General Framework for Communication-Efficient Distributed Optimization
Smith, V., Forte, S., Ma, C., Takáč, M., Jordan, M. I., & Jaggi, M. (2016) · arXiv (Cornell University) · doi:10.3929/ethz-b-000282738 · Cited by 193
Self-citationLocal SGD Converges Fast and Communicates Little
Stich, S. U. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1805.09767 · Cited by 191 · cited 1×
“In response, several recent methods have been proposed to improve communication efficiency in distributed settings by allowing for a variable to be applied on each machine in parallel at each communication round (rather than just computing them locally and then applying them centrally) [44].”
ATOMO: Communication-efficient Learning via Atomic Sparsification
Wang, H., Sievert, S., Charles, Z., Liu, S., Wright, S. J., & Papailiopoulos, D. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1806.04090 · Cited by 189 · cited 1×
“(We refer readers to [47] for a more complete review.”
BackgroundDivide-and-Conquer Matrix FactorizationPreprint
Mackey, L. W., Jordan, M. I., & Talwalkar, A. (2011) · arXiv (Cornell University) · Cited by 182
Self-citationcpSGD: Communication-efficient and differentially-private distributed SGD
Agarwal, N., Suresh, A. T., Yu, F., Kumar, S., & McMahan, H. B. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1805.10559 · Cited by 180 · cited 3×
“Current works that aim to improve the privacy of federated learning typically build upon previous classical cryptographic protocols such as SMC [10, 42] and differential privacy [3, 8, 41, 76].”
BackgroundAIDE: Fast and Communication Efficient Distributed OptimizationPreprint
Reddi, S. J., Konečný, J., Richtárik, P., Póczos, B., & Smola, A. J. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1608.06879 · Cited by 171 · cited 1×
“Several distributed local-updating primal methods have also been proposed, which have the added benefit of being applicable to non-convex objectives [93, 136].”
BackgroundFast Convergence of Stochastic Gradient Descent under a Strong Growth ConditionPreprint
Schmidt, M., & Roux, N. L. (2013) · arXiv (Cornell University) · doi:10.48550/arxiv.1308.6370 · Cited by 171 · cited 1×
“Recent works have aimed to quantify statistical heterogeneity through metrics such as local dissimilarity (as defined in the context of federated learning in [65] and used for other purposes in works such as [100, 116, 130]) and earth mover’s distance [138].”
Backgroundvan Berkel, C. H. (2009) · Design, Automation, and Test in Europe · doi:10.1109/date.2009.5090858 · Cited by 146 · cited 2×
“, millions of smartphones, and communication in the network can be slower than local computation by many orders of magnitude due to limited resources such as bandwidth, energy, and power [46].”
BackgroundCOLA: Decentralized Linear LearningPreprint
He, L., Bian, Y., & Jaggi, M. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1808.04883 · Cited by 125 · cited 3×
“Some recent works have investigated decentralized training over heterogeneous data with local updating schemes [18].”
BackgroundInfluentialGradient Diversity: a Key Ingredient for Scalable Distributed LearningPreprint
Yin, D., Pananjady, A., Lam, M. W. Y., Papailiopoulos, D., Ramchandran, K., & Bartlett, P. L. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1706.05699 · Cited by 119
Zhao, Y., Zhao, J., Jiang, L., Tan, R., & Niyato, D. (2019) · arXiv.org · doi:10.48550/arxiv.1906.10893 · Cited by 115 · cited 1×
“Federated learning methods have been deployed by major service providers [11, 124], and play a critical role in supporting privacy-sensitive applications where the training data are distributed at the edge [e.g., 5, 46, 51, 89, 105, 127, 139].”
MethodDifferentially Private Meta-Learning
Li, J., Khodak, M., Caldas, S., & Talwalkar, A. (2019) · arXiv · doi:10.48550/arxiv.1909.05830 · Cited by 82
Self-citationRobust and communication-efficient collaborative learning
Reisizadeh, A., Taheri, H., Mokhtari, A., Hassani, H., & Pedarsani, R. (2019) · arXiv · doi:10.48550/arxiv.1907.10595 · Cited by 73 · cited 1×
“Some works propose deadlinebased approaches where all workers compute the local gradients using a variable number of samples within a fixed global cycle, which helps mitigate the impact of stragglers [16], [39].”
BackgroundSemi-Cyclic Stochastic Gradient Descent
Eichner, H., Koren, T., McMahan, H. B., Srebro, N., & Talwar, K. (2019) · arXiv · doi:10.48550/arxiv.1904.10120 · Cited by 66 · cited 3×
“[14] investigate a pluralistic solution (adaptively choosing between a global model and device-specific models) to address the cyclic patterns in data samples during federated training.”
BackgroundSecure Computation for Machine Learning With SPDZPreprint
Chen, V., Pastro, V., & Raykova, M. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1901.00329 · Cited by 63 · cited 1×
“Moreover, SMC protocols need to be carefully designed and implemented for each operation in the targeted learning algorithm [25, 79].”
BackgroundScalable and Differentially Private Distributed Aggregation in the Shuffled Model
Ghazi, B., Pagh, R., & Velingker, A. (2019) · arXiv · doi:10.48550/arxiv.1906.08320 · Cited by 62 · cited 7×
“Thus, while SMC cannot guarantee protection from information leakage, it can be combined with differential privacy to achieve stronger privacy guarantees.”
BackgroundInfluentialLoAdaBoost:Loss-Based AdaBoost Federated Machine Learning on medical DataPreprint
Li, H., Yin, Y., Fu, Z., Zhang, S., Deng, H., & Liu, D. (2018) · arXiv (Cornell University) · Cited by 57
Challenges and Solutions of Ubiquitous User Modeling
Kuflik, T., Kay, J., & Kummerfeld, B. (2012) · Cognitive technologies · doi:10.1007/978-3-642-27663-7_2 · Cited by 50 · cited 1×
“Recent works have also considered training machine learning models centrally but serving and storing them locally; for example, this is a common approach in mobile user modeling and personalization [23].”
BackgroundAnytime MiniBatch: Exploiting Stragglers in Online Distributed OptimizationPreprint
Ferdinand, N. S., Al-Lawati, H., Draper, S. C., & Nokleby, M. S. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.05752 · Cited by 49 · cited 1×
“Some works propose deadlinebased approaches where all workers compute the local gradients using a variable number of samples within a fixed global cycle, which helps mitigate the impact of stragglers [16], [39].”
BackgroundDistributed learning over unreliable networks
Yu, C., Tang, H., Renggli, C., Kassing, S., Singla, A., Alistarh, D., Zhang, C., & Liu, J. (2018) · arXiv · doi:10.48550/arxiv.1810.07766 · Cited by 45
δ-Presence without Complete World Knowledge
Nergiz, M. E., & Clifton, C. (2009) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2009.125 · Cited by 39 · cited 1×
“Despite the fact that differential privacy is the de facto metric for privacy in machine learning, there are many other privacy definitions, such as k-anonymity [15] and d-presence [34], which may be applicable to different learning problems.”
BackgroundIncentive Design for Efficient Federated Learning in Mobile Networks: A Contract Theory ApproachPreprint
Kang, J., Xiong, Z., Niyato, D. T., et al. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1905.07479 · Cited by 34 · cited 1×
“[57] take into account”
Client-Edge-Cloud Hierarchical Federated LearningPreprint
Liu, L., Zhang, J., Song, S., et al. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1905.06641 · Cited by 25 · cited 1×
“Finally, hierarchical communication patterns have also been proposed [68, 70] to further ease the burden on the central server, by first leveraging edge servers to aggregate the updates from edge devices and then relying on a cloud server to aggregate updates from edge servers.”
BackgroundGradient Coding Using the Stochastic Block Model
Charles, Z., & Papailiopoulos, D. (2018) · 2018 IEEE International Symposium on Information Theory (ISIT) · doi:10.1109/isit.2018.8437887 · Cited by 23 · cited 2×
“For instance, in the presence of stragglers, gradient coding and its variants [11] carefully replicate data blocks (as well as the gradient computation on those data blocks) across computing nodes to obtain either an exact or inexact recovery of the true gradients.”
BackgroundMLSys: The New Frontier of Machine Learning SystemsPreprint
Ratner, A., Alistarh, D., Alonso, G., Andersen, D. G., Bailis, P. D., Bird, S., Carlini, N., Catanzaro, B., Chayes, J., Chung, E., et al. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1904.03257 · Cited by 20 · cited 1×
“fundamental advances in areas such as privacy, large-scale machine learning, and distributed optimization, and raising new questions at the intersection of diverse fields, such as machine learning and systems [91].”
BackgroundSelf-citationDecentralized Bayesian Learning over Graphs
Lalitha, A., Wang, X., Kilinc, O., Lu, Y., Javidi, T., & Koushanfar, F. (2019) · arXiv · doi:10.48550/arxiv.1905.10466 · Cited by 20 · cited 2×
“Some recent works [47, 61] have investigated decentralized training over heterogeneous data with local updating schemes.”
BackgroundLi, H., & Liu, D. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1903.09296 · Cited by 20
Communication-Efficient Distributed Strongly Convex Stochastic Optimization: Non-Asymptotic RatesPreprint
Sahu, A. K., Jakovetić, D., Bajović, D., & Kar, S. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1809.02920 · Cited by 18
Self-citationLi, H., Yin, Y., Fu, Z., et al. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1811.12629 · Cited by 13
Model Aggregation via Good-Enough Model SpacesPreprint
Guha, N., & Smith, V. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1805.07782 · Cited by 7
Self-citation
Research works printed but not linked: 1
- 0, TensorFlow federated Machine learning on decentralized data
Cited by
Advances and Open Problems in Federated Learning
Kairouz, P., & McMahan, H. B. (2021) · Foundations and Trends in Machine Learning · doi:10.1561/2200000083 · Cited by 7,772
Dwivedi, Y. K., Kshetri, N., Hughes, L., Slade, E. L., Jeyaraj, A., Kar, A. K., Baabdullah, A. M., Koohang, A., Raghavan, V., Ahuja, M., et al. (2023) · International Journal of Information Management · doi:10.1016/j.ijinfomgt.2023.102642 · Cited by 4,584
Federated Optimization in Heterogeneous Networks
Li, T., Sahu, A. K., Zaheer, M., Sanjabi, M., Talwalkar, A., & Smith, V. (2018) · arXiv · doi:10.48550/arxiv.1812.06127 · Cited by 4,023
“While there is a wealth of work on distributed optimization in the context of machine learning, two key challenges distinguish federated learning from traditional distributed optimization: high degrees of systems and statistical heterogeneity1 (McMahan et al., 2017; Li et al., 2019).”
BackgroundSelf-citationThe future of digital health with federated learning
Rieke, N., Hancox, J., Li, W., Milletarì, F., Roth, H. R., Albarqouni, S., Bakas, S., Galtier, M. N., Landman, B. A., Maier-Hein, K., et al. (2020) · npj Digital Medicine · doi:10.1038/s41746-020-00323-1 · Cited by 4,015
“However, there is a trade-off in terms of performance and these techniques may affect for example the accuracy of the final model [10].”
BackgroundMethodThe Future of Digital Health with Federated Learning
Rieke, N., Hancox, J., Li, W., Milletari, F., Roth, H., Albarqouni, S., Bakas, S., Galtier, M. N., Landman, B., Maier-Hein, K., et al. (2020) · arXiv · doi:10.48550/arxiv.2003.08119 · Cited by 3,495
“However, there is a trade-off in terms of performance and these techniques may affect for example the accuracy of the final model [10].”
BackgroundMethod
Metadata
- Authors
- Li, T., Sahu, A. K., Talwalkar, A., & Smith, V.
- Affiliations
- Carnegie Mellon University (US)
- ORCID
- Sahu, Anit Kumar
- Venue
- IEEE Signal Processing Magazine
- Published
- 2020-05
- Volume
- 37
- Issue
- 3
- Pages
- 50-60
- ISSN
- 1053-5888 (ISSN-L)
- 1558-0792
- OpenAlex
- W3021654819
- Cited by
- 6,210
- References
- 53
- Citations by year
- 2026: 955 · 2025: 1259 · 2024: 1180 · 2023: 1143 · 2022: 789 · 2021: 624 · 2020: 233
- FWCI
- 274.11
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- IEEE Signal Processing Magazine (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Privacy-Preserving Technologies in Data
- Physical Sciences › Computer Science › Artificial Intelligence › Stochastic Gradient Optimization Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Keywords
- federated learning, data localization, privacy-preserving machine learning, distributed optimization, heterogeneous networks, large-scale machine learning, mobile devices
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.