CoChat
Sign Up
PreprintCited by 894Privacy-Preserving Technologies in DataDomain Adaptation and Few-Shot LearningMachine Learning and Data Classification

LEAF: A Benchmark for Federated Settings

Open access
Caldas, Sebastian, Duddu, Sai Meher Karthik, Wu, Peter, Li, Tian, Konečný, Jakub, McMahan, H. Brendan, Smith, Virginia, Talwalkar, Ameet·2018·arXiv
DOI 10.48550/arxiv.1812.01097arXiv 1812.01097
Open this paper in CoChat — free accountOpen PDF

Abstract

Modern federated networks, such as those comprised of wearable devices, mobile phones, or autonomous vehicles, generate massive amounts of data each day. This wealth of data can help to learn models that can improve the user experience on each device. However, the scale and heterogeneity of federated data presents new challenges in research areas such as federated learning, meta-learning, and multi-task learning. As the machine learning community begins to tackle these challenges, we are at a critical time to ensure that developments made in these areas are grounded with realistic benchmarks. To this end, we propose LEAF, a modular benchmarking framework for learning in federated settings. LEAF includes a suite of open-source federated datasets, a rigorous evaluation framework, and a set of reference implementations, all geared towards capturing the obstacles and intricacies of practical federated environments.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.1812.01097

Cites

6 of 6 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Because clients cannot all be guaranteed reliable, federated learning needs extra privacy protections for unreliable participants.

    “Privacy and Security: Additional privacy protections are needed for unreliable participating clients. It is impossible to ensure all clients are equally reliable.”

    From Federated Learning for Healthcare Informatics (2021) · cites this paper as [13]

    The Introduction (p. 1) says federated methods must address privacy and security concerns because personally generated data is sensitive. It does not say that participating clients are unreliable, that their reliability cannot be ensured, or that unreliability calls for additional privacy protections. (LLM, conf=0.83)

    Experimental — accuracy not yet measured

  • Client data distributions in federated learning are highly different, so local samples do not represent the overall distribution.

    “Statistical Challenge: The data distribution among all clients differ greatly, i.e., , we have . It is such that any data points available locally are far from being a representative sample of the overall distribution, i.e., .”

    From Federated Learning for Healthcare Informatics (2021) · cites this paper as [13]

    The Introduction (p. 1) says devices have different, though perhaps related, data-generating distributions, supporting the claim of heterogeneous client data. It does not establish that every client's local samples are far from representative of the overall distribution; that absolute formulation goes beyond the passage. (LLM, conf=0.84)

    Experimental — accuracy not yet measured

  • We evaluate models by their accuracy on the reviewer at the 10th percentile, following the federated learning literature.

    “To assess whether models perform consistently well across reviewers, we evaluate models by their accuracy on the reviewer at the 10th percentile. This follows the federated learning literature, where it is standard to measure model performance on devices and users at various percentiles in an effort to encourage good performance across many devices (Caldas et al., 2018; Li et al., 2019b).”

    From WILDS: A Benchmark of in-the-Wild Distribution Shifts (2020) · cites this paper as Caldas et al., 2018

    The LEAF section (p. 3) introduces performance at the 10th, 50th, and 90th percentiles to capture the distribution across devices, supporting the choice of a 10th-percentile metric. It does not establish that measuring performance at various percentiles was already standard in federated-learning literature; it describes these as metrics LEAF introduces. (LLM, conf=0.86)

    Experimental — accuracy not yet measured

  • Federated learning often involves many clients, and the client count can exceed the average number of training samples per client.

    “Communication Efficiency: The number of clients K is large and can be much bigger than the average number of training sample stored in the activated clients, i.e., K ≫ (n/K).”

    From Federated Learning for Healthcare Informatics (2021) · cites this paper as [13]

  • Models can perform well on average but poorly on some users.

    “These models can perform well on average but poorly on some users (Tatman, 2017; Caldas et al., 2018; Li et al., 2019b; Koenecke et al., 2020).”

    From WILDS: A Benchmark of in-the-Wild Distribution Shifts (2020) · cites this paper as Caldas et al., 2018

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 27

  1. [1][1] Arvind Agarwal, Samuel Gerber, and Hal Daume. Learning multiple tasks using manifold regularization. In Advances in neural information processing systems , 2010. -
  2. [2][2] Andreas Argyriou, Theodoros Evgeniou, and Massimiliano Pontil. Convex multi-task feature learning. Machine Learning , 73(3):243-272, 2008. -
  3. [4][4] Keith Bonawitz, Vladimir Ivanov, Ben Kreuter, Antonio Marcedone, H Brendan McMahan, Sarvar Patel, Daniel Ramage, Aaron Segal, and Karn Seth. Practical secure aggregation for privacy-preserving machine learning. In ACM SIGSAC Conference on Computer and Communications Security , 2017. -
  4. [5][5] Fei Chen, Zhenhua Dong, Zhenguo Li, and Xiuqiang He. Federated meta-learning for recommendation. arXiv preprint arXiv:1802.07876 , 2018. - · arXiv
  5. [6][6] Gregory Cohen, Saeed Afshar, Jonathan Tapson, and André van Schaik. EMNIST: an extension of MNIST to handwritten letters. arXiv preprint arXiv:1702.05373 , 2017. - · arXiv
  6. [7][7] Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In International Conference on Machine Learning , 2017. -
  7. [8][8] Robin C Geyer, Tassilo Klein, and Moin Nabi. Differentially private federated learning: A client level perspective. arXiv preprint arXiv:1712.07557 , 2017. - · arXiv
  8. [11][11] Michael Kamp, Linara Adilova, Joachim Sicking, Fabian Hüger, Peter Schlicht, Tim Wirtz, and Stefan Wrobel. Efficient decentralized deep learning by dynamic model averaging. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases , 2018. -
  9. [12][12] Mikhail Khodak, Maria Florina-Balcan, and Ameet Talwalkar. Adaptive gradient-based metalearning methods. Advances in Neural Information Processing Systems , 2019. -
  10. [13][13] Jakub Koneˇ cn` y, H Brendan McMahan, Felix X Yu, Peter Richtárik, Ananda Theertha Suresh, and Dave Bacon. Federated learning: Strategies for improving communication efficiency. arXiv preprint arXiv:1610.05492 , 2016. - · arXiv
  11. [14][14] Abhishek Kumar and Hal Daume III. Learning task grouping and overlap in multi-task learning. arXiv preprint arXiv:1206.6417 , 2012. - · arXiv
  12. [17][17] Giwoong Lee, Eunho Yang, and Sung Hwang. Asymmetric multi-task learning based on task relatedness and loss. In International Conference on Machine Learning , 2016. -
  13. [19][19] Tian Li, Anit Kumar Sahu, Ameet Talwalkar, and Virginia Smith. Federated learning: Challenges, methods, and future directions. arXiv preprint arXiv:1908.07873 , 2019. - · arXiv
  14. [20][20] Tian Li, Maziar Sanjabi, and Virginia Smith. Fair resource allocation in federated learning. arXiv preprint arXiv:1905.10497 , 2019. - · arXiv
  15. [21][21] Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In International Conference on Computer Vision , 2015. -
  16. [25][25] Luca Melis, Congzheng Song, Emiliano De Cristofaro, and Vitaly Shmatikov. Inference attacks against collaborative learning. arXiv preprint arXiv:1805.04049 , 2018. - · arXiv
  17. [26][26] Keerthiram Murugesan and Jaime Carbonell. Multi-task multiple kernel relationship learning. In SIAM International Conference on Data Mining , 2017. -
  18. [28][28] Vasyl Pihur, Aleksandra Korolova, Frederick Liu, Subhash Sankuratripati, Moti Yung, Dachuan Huang, and Ruogu Zeng. Differentially-private "draw and discard" machine learning. arXiv preprint arXiv:1807.04369 , 2018. - · arXiv
  19. [29][29] Sachin Ravi and Hugo Larochelle. Optimization as a model for few-shot learning. https: // openreview. net/ pdf? id= rJY0-Kcll , 2016. -
  20. [30][30] Virginia Smith, Chao-Kai Chiang, Maziar Sanjabi, and Ameet S Talwalkar. Federated multi-task learning. In Advances in Neural Information Processing Systems , 2017. -
  21. [31][31] Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. In Advances in Neural Information Processing Systems , 2017. -
  22. [32][32] Gregor Ulm, Emil Gustavsson, and Mats Jirstrand. Functional federated learning in erlang (ffl-erl). In International Workshop on Functional and Constraint Logic Programming , 2018. -
  23. [33][33] Oriol Vinyals, Charles Blundell, Tim Lillicrap, Daan Wierstra, et al. Matching networks for one shot learning. In Advances in Neural Information Processing Systems , 2016. -
  24. [34][34] Shiqiang Wang, Tiffany Tuor, Theodoros Salonidis, Kin K Leung, Christian Makaya, Ting He, and Kevin Chan. Adaptive federated learning in resource constrained edge computing systems. IEEE Journal on Selected Areas in Communications , 2019. -
  25. [36][36] Ya Xue, Xuejun Liao, Lawrence Carin, and Balaji Krishnapuram. Multi-task learning for classification with dirichlet process priors. Journal of Machine Learning Research , 8(Jan):3563, 2007. -
  26. [37][37] Qiang Yang, Yang Liu, Tianjian Chen, and Yongxin Tong. Federated machine learning: Concept and applications. ACM Transactions on Intelligent Systems and Technology , 2019. -
  27. [38][38] Yi Zhang and Jeff G Schneider. Learning multiple tasks with a sparse matrix-normal penalty. In Advances in Neural Information Processing Systems , 2010.

Cited by

Sorted by citations

Authors
Caldas, S., Duddu, S. M. K., Wu, P., Li, T., Konečný, J., McMahan, H. B., Smith, V., & Talwalkar, A.
Affiliations
  • Carnegie Mellon University (US)
  • Google (United States) (US)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2018
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.1812.01097
arXiv
1812.01097
OpenAlex
W2902113386
Cited by
894
References
6
Citations by year
2026: 6 · 2025: 72 · 2024: 197 · 2023: 186 · 2022: 146 · 2021: 175 · 2020: 82 · 2019: 28 · 2018: 2
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Funders
  • National Science Foundation
  • Semiconductor Research Corporation
  • Amazon Web Services
  • Defense Advanced Research Projects Agency
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Privacy-Preserving Technologies in Data
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Data Classification
Keywords
federated learning, benchmarking framework, data heterogeneity, meta-learning, multi-task learning, wearable devices, mobile devices, autonomous vehicles, evaluation framework, reference implementation

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review