CoChat

Federated Learning: Challenges, Methods, and Future Directions

Open access
Li, Tian, Sahu, Anit Kumar, Talwalkar, Ameet, Smith, Virginia · 2020 · IEEE Signal Processing Magazine

The DOI matches its Crossref record; 92 of the 92 references it cites were found in a registry or an index.

Abstract

Federated learning involves training statistical models over remote devices or siloed data centers, such as mobile phones or hospitals, while keeping data localized. Training in heterogeneous and potentially massive networks introduces novel challenges that require a fundamental departure from standard approaches for large-scale machine learning, distributed optimization, and privacy-preserving data analysis. In this article, we discuss the unique characteristics and challenges of federated learning, provide a broad overview of current approaches, and outline several directions of future work that are relevant to a wide range of research communities.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1109/msp.2020.2975749

Cites

92 of 92 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

2 more claims about this paper

Create a free account to see every claim other papers make about this one.

See all claims — free account

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 92 references

Research works printed but not linked: 56

  1. [77] L. Melis, G. Danezis, and E. D. Cristofaro. Efficient private statistics with succinct sketches. In Network and Distributed System Security Symposium , 2016. -
  2. [79] P. Mohassel and P. Rindal. Aby 3: a mixed protocol framework for machine learning. In Conference on Computer and Communications Security , 2018. -
  3. [80] M. Mohri, G. Sivek, and A. T. Suresh. Agnostic federated learning. In International Conference on Machine Learning , 2019. -
  4. [94] A. Reisizadeh, S. Prakash, R. Pedarsani, and A. S. Avestimehr. Coded computation over heterogeneous clusters. IEEE Transactions on Information Theory , 65:4227-4242, 2019. -
  5. [95] M. S. Riazi, C. Weinert, O. Tkachenko, E. M. Songhori, T. Schneider, and F. Koushanfar. Chameleon: A hybrid secure computation framework for machine learning applications. In Asia Conference on Computer and Communications Security , 2018. -
  6. [96] P. Richtárik and M. Takᡠc. Distributed coordinate descent method for learning with big data. Journal of Machine Learning Research , 17:2657-2681, 2016. -
  7. [97] B. D. Rouhani, M. S. Riazi, and F. Koushanfar. Deepsecure: Scalable provably-secure deep learning. In Design Automation Conference , 2018. -
  8. [98] S. Samarakoon, M. Bennis, W. Saad, and M. Debbah. Federated learning for ultra-reliable low-latency v2v communications. In Global Communications Conference , 2018. -
  9. [102] S. Shalev-Shwartz and T. Zhang. Accelerated mini-batch stochastic dual coordinate ascent. In Advances in Neural Information Processing Systems , 2013. -
  10. [103] O. Shamir and N. Srebro. Distributed stochastic optimization and learning. In Allerton Conference on Communication, Control, and Computing , 2014. -
  11. [105] S. Silva, B. Gutman, E. Romero, P. M. Thompson, A. Altmann, and M. Lorenzi. Federated learning in distributed medical databases: Meta-analysis of large-scale subcortical brain data. arXiv preprint arXiv:1810.08553 , 2018. - · arXiv
  12. [86] N. Papernot, S. Song, I. Mironov, A. Raghunathan, K. Talwar, and Ú. Erlingsson. Scalable private learning with pate. In International Conference on Learning Representations , 2018. -
  13. [87] A. Qiao, B. Aragam, B. Zhang, and E. Xing. Fault tolerance in iterative-convergent machine learning. In International Conference on Machine Learning , 2019. -
  14. [88] Z. Qu, P. Richtárik, and T. Zhang. Quartz: Randomized dual coordinate ascent with arbitrary sampling. In Advances in Neural Information Processing Systems , 2015. -
  15. [71] Y. Liu, J. K. Muppala, M. Veeraraghavan, D. Lin, and M. Hamdi. Data center networks: Topologies, architectures and fault-tolerance characteristics . Springer Science & Business Media, 2013. -
  16. [72] C. Ma, V. Smith, M. Jaggi, M. I. Jordan, P. Richtárik, and M. Takᡠc. Adding vs. averaging in distributed primal-dual optimization. In International Conference on Machine Learning , 2015. -
  17. [6] D. Anguita, A. Ghio, L. Oneto, X. Parra, and J. L. Reyes-Ortiz. A public domain dataset for human activity recognition using smartphones. In European Symposium on Artificial Neural Networks, Computational Intelligence and Machine Learning , 2013. -
  18. [56] P. Jiang and G. Agrawal. A linear speedup analysis of distributed deep learning with sparse and quantized communication. In Advances in Neural Information Processing Systems , 2018. -
  19. [62] C.-P. Lee and D. Roth. Distributed box-constrained quadratic optimization for dual linear svm. In International Conference on Machine Learning , 2015. -
  20. [63] K. Lee, M. Lam, R. Pedarsani, D. Papailiopoulos, and K. Ramchandran. Speeding up distributed machine learning using codes. IEEE Transactions on Information Theory , 64:1514-1529, 2017. -
  21. [131] H. Yu, S. Yang, and S. Zhu. Parallel restarted sgd for non-convex optimization with faster convergence and less communication. In AAAI Conference on Artificial Intelligence , 2018. -
  22. [132] H. Yu, R. Jin, and S. Yang. On the linear speedup analysis of communication efficient momentum sgd for distributed non-convex optimization. In International Conference on Machine Learning , 2019. -
  23. [133] J. Yuan and S. Yu. Privacy preserving back-propagation neural network learning made practical with cloud computing. IEEE Transactions on Parallel and Distributed Systems , 25:212-221, 2013. -
  24. [134] M. Yurochkin, M. Agarwal, S. Ghosh, K. Greenewald, T. N. Hoang, and Y. Khazaeni. Bayesian nonparametric federated learning of neural networks. In International Conference on Machine Learning , 2019. -
  25. [135] H. Zhang, J. Li, K. Kara, D. Alistarh, J. Liu, and C. Zhang. ZipML: Training linear models with end-to-end low precision, and a little bit of deep learning. In International Conference on Machine Learning , 2017. -
  26. [1] Tensorflow federated: Machine learning on decentralized data. URL https://www.tensorflow.org/federated . -
  27. [2] M. Abadi, A. Chu, I. Goodfellow, H. B. McMahan, I. Mironov, K. Talwar, and L. Zhang. Deep learning with differential privacy. In Conference on Computer and Communications Security , 2016. -
  28. [137] Y. Zhang, J. Duchi, and M. Wainwright. Divide and conquer kernel ridge regression: A distributed algorithm with minimax optimal rates. Journal of Machine Learning Research , 16:3299-3340, 2015. -
  29. [140] F. Zhou and G. Cong. On the convergence properties of a k -step averaging stochastic gradient descent algorithm for nonconvex optimization. In International Joint Conference on Artificial Intelligence , 2018. -
  30. [141] M. Zinkevich, M. Weimer, L. Li, and A. J. Smola. Parallelized stochastic gradient descent. In Advances in Neural Information Processing Systems , 2010.
  31. [21] Z. B. Charles, D. S. Papailiopoulos, and J. Ellenberg. Approximate gradient coding via sparse random graphs. arXiv preprint arXiv:1711.0677 , 2017. -
  32. [22] K. Chaudhuri, C. Monteleoni, and A. D. Sarwate. Differentially private empirical risk minimization. Journal of Machine Learning Research , 12:1069-1109, 2011. -
  33. [23] D. Chaum. The dining cryptographers problem: Unconditional sender and recipient untraceability. Journal of Cryptology , 1:65-75, 1988. -
  34. [27] W. Dai, A. Kumar, J. Wei, Q. Ho, G. Gibson, and E. P. Xing. High-performance distributed ML at scale through parameter server consistency models. In AAAI Conference on Artificial Intelligence , 2015. -
  35. [28] O. Dekel, R. Gilad-Bachrach, O. Shamir, and L. Xiao. Optimal distributed online prediction using mini-batches. Journal of Machine Learning Research , 13:165-202, 2012. -
  36. [29] A. Deshpande, C. Guestrin, S. R. Madden, J. M. Hellerstein, and W. Hong. Model-based approximate querying in sensor networks. The VLDB Journal , 14:417-443, 2005. -
  37. [30] J. Duchi, M. I. Jordan, and B. McMahan. Estimation, optimization, and parallelism when data is sparse. In Advances in Neural Information Processing Systems , 2013. -
  38. [31] J. C. Duchi, M. I. Jordan, and M. J. Wainwright. Privacy aware learning. In Advances in Neural Information Processing Systems , 2012. -
  39. [32] C. Dwork. A firm foundation for private data analysis. Communications of the ACM , 54:86-95, 2011. -
  40. [37] T. Evgeniou and M. Pontil. Regularized multi-task learning. In Conference on Knowledge Discovery and Data Mining , 2004. -
  41. [39] M. Fredrikson, S. Jha, and T. Ristenpart. Model inversion attacks that exploit confidence information and basic countermeasures. In Conference on Computer and Communications Security , 2015. -
  42. [43] S. Goryczka and L. Xiong. A comprehensive comparison of multiparty secure additions with differential privacy. IEEE Transactions on Dependable and Secure Computing , 14:463-477, 2015. -
  43. [48] Q. Ho, J. Cipar, H. Cui, S. Lee, J. K. Kim, P. B. Gibbons, G. A. Gibson, G. Ganger, and E. P. Xing. More effective distributed ML via a stale synchronous parallel parameter server. In Advances in Neural Information Processing Systems , 2013. -
  44. [50] J. Huang, F. Qian, Y. Guo, Y. Zhou, Q. Xu, Z. M. Mao, S. Sen, and O. Spatscheck. An in-depth study of lte: effect of network protocol and application behavior on performance. SIGCOMM Computer Communication Review , 43: 363-374, 2013. -
  45. [53] R. Iyengar, J. P. Near, D. Song, O. Thakkar, A. Thakurta, and L. Wang. Towards practical differentially private convex optimization. In Conference on Computer and Communications Security , 2019. -
  46. [54] M. Jaggi, V. Smith, M. Takác, J. Terhorst, S. Krishnan, T. Hofmann, and M. I. Jordan. Communication-efficient distributed dual coordinate ascent. In Advances in Neural Information Processing Systems , 2014. -
  47. [116] S. Vaswani, F. Bach, and M. Schmidt. Fast and faster convergence of sgd for over-parameterized models (and an accelerated perceptron). In Conference on Artificial Intelligence and Statistics , 2019. -
  48. [118] I. Wagner and D. Eckhoff. Technical privacy metrics: a systematic survey. ACM Computing Surveys , 51:57, 2018. -
  49. [114] S. Thrun and L. Pratt. Learning to learn . Springer Science & Business Media, 2012. -
  50. [121] J. Wang and G. Joshi. Adaptive communication strategies to achieve the best error-runtime trade-off in localupdate sgd. In Conference on Systems and Machine Learning , 2019. -
  51. [122] S. Wang, F. Roosta-Khorasani, P. Xu, and M. W. Mahoney. Giant: Globally improved approximate newton method for distributed optimization. In Advances in Neural Information Processing Systems , 2018. -
  52. [125] B. Woodworth, J. Wang, A. Smith, B. McMahan, and N. Srebro. Graph oracle models, lower bounds, and gaps for parallel stochastic optimization. In Advances in Neural Information Processing Systems , 2018. -
  53. [126] X. Wu, F. Li, A. Kumar, K. Chaudhuri, S. Jha, and J. Naughton. Bolt-on differential privacy for scalable stochastic gradient descent-based analytics. In International Conference on Management of Data , 2017. -
  54. [128] T. Yang. Trading computation for communication: Distributed stochastic dual coordinate ascent. In Advances in Neural Information Processing Systems , 2013. -
  55. [109] R. Tandon, Q. Lei, A. G. Dimakis, and N. Karampatziakis. Gradient coding: Avoiding stragglers in distributed learning. In International Conference on Machine Learning , 2017. -
  56. [107] V. Smith, S. Forte, C. Ma, M. Takac, M. I. Jordan, and M. Jaggi. Cocoa: a general framework for communicationefficient distributed optimization. Journal of Machine Learning Research , 18:1-47, 2018. -

Cited by

Items 101–200 of 1,043

Metadata

Authors
Li, T., Sahu, A. K., Talwalkar, A., & Smith, V.
Affiliations
Carnegie Mellon University (US)
Venue
IEEE Signal Processing Magazine
Published
2020-05
Volume
37
Issue
3
Pages
50-60
ISSN
  • 1053-5888 (ISSN-L)
  • 1558-0792
OpenAlex
W3021654819
Cited by
6,210
References
53
Citations by year
2026: 955 · 2025: 1259 · 2024: 1180 · 2023: 1143 · 2022: 789 · 2021: 624 · 2020: 233
FWCI
274.11
Type
journal article
Language
en
Open access
Locations
  • IEEE Signal Processing Magazine (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Privacy-Preserving Technologies in Data
  • Physical Sciences › Computer Science › Artificial Intelligence › Stochastic Gradient Optimization Techniques
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
Keywords
federated learning, data localization, privacy-preserving machine learning, distributed optimization, heterogeneous networks, large-scale machine learning, mobile devices

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.