CoChat
Sign Up
PreprintCited by 2,334Privacy-Preserving Technologies in DataStochastic Gradient Optimization TechniquesDomain Adaptation and Few-Shot Learning

Towards Federated Learning at Scale: System Design

Open access
Bonawitz, Keith, Eichner, Hubert, Grieskamp, Wolfgang, Huba, Dzmitry, Ingerman, Alex, Ivanov, Vladimir, Kiddon, Chloe, Konečný, Jakub +6 more·2019·arXiv
DOI 10.48550/arxiv.1902.01046arXiv 1902.01046
Open this paper in CoChat — free accountOpen PDF

Abstract

Federated Learning is a distributed machine learning approach which enables model training on a large corpus of decentralized data. We have built a scalable production system for Federated Learning in the domain of mobile devices, based on TensorFlow. In this paper, we describe the resulting high-level design, sketch some of the challenges and their solutions, and touch upon the open problems and future directions.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.1902.01046

Cites

6 of 6 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • When the number of clients grows exponentially, direct communications between numerous clients and a server for offloading updates become infeasible due to increasing workload on network channels.

    “Further, when the number of clients grows exponentially, direct communications between numerous clients and a server for offloading updates become infeasible due to the increasing workload on the network channels [93].”

    From Federated Learning for Internet of Things: A Comprehensive Survey (2021) · cites this paper as [93]

    Section 4 (p. 4) describes the challenge of scaling server communication across very large device populations and variable traffic; Section 4.2 (p. 5) describes delegating work to aggregators as device counts and update sizes grow. The paper does not say client numbers grow exponentially or that direct client–server communication becomes infeasible because network channels are overloaded. (LLM, conf=0.85)

    Experimental — accuracy not yet measured

  • [199] improves scalability up to 1.5e6 clients.

    “Scalability improvement: up to 1.5e6 clients”

    From Convergence of Edge Computing and Deep Learning: A Comprehensive Survey (2020) · cites this paper as [199]

    The Operational Profile (p. 9) supports the general scalability claim: the system handled approximately 10 million daily active devices. It does not report 1.5 million clients as a scalability limit or measured result; it separately reports up to 10,000 devices participating simultaneously. (LLM, conf=0.85)

    Experimental — accuracy not yet measured

  • Existing FL methods focus on synchronous training, and can only process hundreds of devices in parallel.

    “Existing FL methods [198], [199] focus on synchronous training, and can only process hundreds of devices in parallel.”

    From Convergence of Edge Computing and Deep Learning: A Comprehensive Survey (2020) · cites this paper as [199]

    The Introduction (p. 1) says the authors chose to support synchronous rounds, and Future Work (p. 10) says current FL algorithms can efficiently utilize only hundreds of devices in parallel. The claim drops the efficiency qualifier: the Operational Profile (p. 9) reports up to 10,000 devices participating simultaneously, so the paper does not establish a hard limit of hundreds that can be processed. (LLM, conf=0.87)

    Experimental — accuracy not yet measured

  • FL only copies the whole DL model to every participated edge device, namely in the manner of data parallelism.

    “At present, FL only copies the whole DL model to every participated edge devices, namely in the manner of data parallelism.”

    From Convergence of Edge Computing and Deep Learning: A Comprehensive Survey (2020) · cites this paper as [199]

    Sections 2.1–2.2 (pp. 2–3) describe the server sending the global model to each selected device, which then computes an update using its local data. That supports the data-parallel description of this system, but the paper does not establish the broader claim that FL *only* works this way. (LLM, conf=0.86)

    Experimental — accuracy not yet measured

  • Federated learning is promising for natural language processing tasks such as virtual keyboard prediction.

    “Therefore, federated learning has a promising application in NLP like virtual keyboard prediction [7, 39, 70].”

    From Federated Learning for Healthcare Informatics (2021) · cites this paper as [7]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 18

  1. Abadi, M., Agarwal, A., Barham, P., Brevdo, E., Chen, Z., Citro, C., Corrado, G. S., Davis, A., Dean, J., Devin, M., Ghemawat, S., Goodfellow, I., Harp, A., Irving, G., Isard, M., Jia, Y., Jozefowicz, R., Kaiser, L., Kudlur, M., Levenberg, J., Man´ e, D., Monga, R., Moore, S., Murray, D., Olah, C., Schuster, M., Shlens, J., Steiner, B., Sutskever, I., Talwar, K., Tucker, P., Vanhoucke, V., Vasudevan, V., Vi´ egas, F., Vinyals, O., Warden, P., Wattenberg, M., Wicke, M., Yu, Y., and Zheng, X. TensorFlow: Largescale machine learning on heterogeneous systems. In OSDI , volume 16, pp. 265-283, 2016. ai.google. Under the hood of the pixel 2: How ai is supercharging hardware, 2018. URL https://ai. google/stories/ai-in-hardware/ . Retrieved Nov 2018.
  2. Bonawitz, K., Ivanov, V., Kreuter, B., Marcedone, A., McMahan, H. B., Patel, S., Ramage, D., Segal, A., and Seth, K. Practical secure aggregation for privacypreserving machine learning. In Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security , pp. 1175-1191. ACM, 2017.
  3. Dean, J. and Ghemawat, S. MapReduce: Simplified data processing on large clusters. Communications of the ACM , 51(1):107-113, 2008.
  4. Dean, J., Corrado, G., Monga, R., Chen, K., Devin, M., Le, Q. V., Mao, M., Ranzato, M., Senior, A., Tucker, P., Yang, K., and Ng, A. Y. Large scale distributed deep networks. In Advances in neural information processing systems , pp. 1223-1231, 2012.
  5. Goyal, P., Doll´ ar, P., Girshick, R., Noordhuis, P., Wesolowski, L., Kyrola, A., Tulloch, A., Jia, Y., and He, K. Accurate, large minibatch SGD: Training imagenet in 1 hour. arXiv preprint arXiv:1706.02677 , 2017. · arXiv
  6. Hard, A., Rao, K., Mathews, R., Beaufays, F., Augenstein, S., Eichner, H., Kiddon, C., and Ramage, D. Federated learning for mobile keyboard prediction. arXiv preprint 1811.03604 , 2018. · arXiv
  7. Hewitt, C., Bishop, P. B., and Steiger, R. A universal modular ACTOR formalism for artificial intelligence. In Proceedings of the 3rd International Joint Conference on Artificial Intelligence. Stanford, CA, USA, August 20-23, 1973 , pp. 235-245, 1973.
  8. Jacob, B., Kligys, S., Chen, B., Zhu, M., Tang, M., Howard, A., Adam, H., and Kalenichenko, D. Quantization and training of neural networks for efficient integer-arithmetic-only inference. arXiv preprint arXiv:1712.05877 , 2017. · arXiv
  9. Kamp, M., Adilova, L., Sicking, J., H¨ uger, F., Schlicht, P., Wirtz, T., and Wrobel, S. Efficient decentralized deep learning by dynamic model averaging. arXiv preprint arXiv:1807.03210 , 2018. · arXiv
  10. Koneˇ cn´ y, J., McMahan, H. B., Ramage, D., and Richt´ arik, P. Federated optimization: Distributed machine learning for on-device intelligence. arXiv preprint arXiv:1610.02527 , 2016a. · arXiv
  11. Koneˇ cn´ y, J., McMahan, H. B., Yu, F. X., Richt´ arik, P., Suresh, A. T., and Bacon, D. Federated learning: Strategies for improving communication efficiency. arXiv preprint arXiv:1610.05492 , 2016b. · arXiv
  12. Li, M., Andersen, D. G., Park, J. W., Smola, A. J., Ahmed, A., Josifovski, V., Long, J., Shekita, E. J., and Su, B.-Y . Scaling distributed machine learning with the parameter server. In 11th USENIX Symposium on Operating Systems Design and Implementation (OSDI 14) , pp. 583-598. USENIX Association, 2014.
  13. Low, Y., Bickson, D., Gonzalez, J., Guestrin, C., Kyrola, A., and Hellerstein, J. M. Distributed graphlab: A framework for machine learning and data mining in the cloud. Proc. VLDB Endow. , 5(8):716-727, April 2012.
  14. Nishio, T. and Yonetani, R. Client selection for federated learning with heterogeneous resources in mobile edge. arXiv preprint arXiv:1804.08333 , 2018. · arXiv
  15. Pihur, V., Korolova, A., Liu, F., Sankuratripati, S., Yung, M., Huang, D., and Zeng, R. Differentially-private 'draw and discard' machine learning. arXiv preprint arXiv:1807.04369 , 2018. · arXiv
  16. Smith, S., jan Kindermans, P., Ying, C., and Le, Q. V. Don't decay the learning rate, increase the batch size. In International Conference on Learning Representations (ICLR) , 2018.
  17. Smith, V., Chiang, C.-K., Sanjabi, M., and Talwalkar, A. S. Federated multi-task learning. In Advances in Neural Information Processing Systems , pp. 4424-4434, 2017.
  18. Yang, T., Andrew, G., Eichner, H., Sun, H., Li, W., Kong, N., Ramage, D., and Beaufays, F. Applied federated learning: Improving google keyboard query suggestions. arXiv preprint 1812.02903 , 2018. · arXiv

Cited by

Sorted by citations

Authors
Bonawitz, K., Eichner, H., Grieskamp, W., Huba, D., Ingerman, A., Ivanov, V., Kiddon, C., Konečný, J., Mazzocchi, S., McMahan, H. B., Van Overveldt, T., Petrou, D., Ramage, D., & Roselander, J.
ORCID
Venue
arXiv
Publisher
arXiv
Published
2019
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.1902.01046
arXiv
1902.01046
OpenAlex
W2913570153
Cited by
2,334
References
6
Citations by year
2026: 9 · 2025: 174 · 2024: 426 · 2023: 457 · 2022: 382 · 2021: 478 · 2020: 321 · 2019: 81 · 2018: 6
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Privacy-Preserving Technologies in Data
  • Physical Sciences › Computer Science › Artificial Intelligence › Stochastic Gradient Optimization Techniques
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
Keywords
federated learning, production system design, scalability, mobile devices, decentralized data, distributed learning, TensorFlow

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review