Federated Learning with Non-IID Data
Open accessAbstract
Federated learning enables resource-constrained edge compute devices, such as mobile phones and IoT devices, to learn a shared model for prediction, while keeping the training data local. This decentralized approach to train models provides privacy, security, regulatory and economic benefits. In this work, we focus on the statistical challenge of federated learning when local data is non-IID. We first show that the accuracy of federated learning reduces significantly, by up to 55% for neural networks trained for highly skewed non-IID data, where each client device trains only on a single class of data. We further show that this accuracy reduction can be explained by the weight divergence, which can be quantified by the earth mover's distance (EMD) between the distribution over classes on each device and the population distribution. As a solution, we propose a strategy to improve training on non-IID data by creating a small subset of data which is globally shared between all the edge devices. Experiments show that accuracy can be increased by 30% for the CIFAR-10 dataset with only 5% globally shared data.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Non-IID data skews local client updates, degrading global model convergence and fairness.
“Non-IID data skews local client updates, degrading global model convergence and fairness.”
From Privacy-preserving Machine Learning in Internet of Vehicle Applications: Fundamentals, Recent Advances, and Future Direction (2025) · cites this paper as [192]
Section 3.1 (p. 4) says non-IID client data makes local weights diverge more from centralized weights, and Section 2.2 (p. 3) reports reduced FedAvg test accuracy. These passages support the claim about skewed local updates and degraded model performance, but they do not establish degraded convergence as distinct from accuracy, or address fairness. (LLM, conf=0.86)
Experimental — accuracy not yet measuredFL's iterative model update mechanism introduces communication inefficiencies, particularly in large-scale deployments with non-IID data distributions.
“FL's iterative model update mechanism introduces com munication inefficiencies, particularly in largescale deployments with non-IID data distributions [192].”
From Privacy-preserving Machine Learning in Internet of Vehicle Applications: Fundamentals, Recent Advances, and Future Direction (2025) · cites this paper as [192]
The Introduction (p. 1) identifies communication costs from transferring model weights and synchronization latency, supporting the general point that federated learning has communication inefficiencies. But it treats non-IID data as a separate statistical challenge; the cited passages do not establish that communication inefficiencies are particularly severe in large-scale deployments with non-IID data. (LLM, conf=0.86)
Experimental — accuracy not yet measuredCentralized models trained on pooled local samples minimize loss against a uniform distribution.
“Most centralized models are trained on the aggregated training samples obtained from the samples drawn from the local clients [96, 111]. Intrinsically, the centralized model is trained to minimize the loss with respect to the uniform distribution [73]”
From Federated Learning for Healthcare Informatics (2021) · cites this paper as [111]
Section 3.1 (p. 4) describes centralized training as optimizing population loss, but it does not say the population distribution is uniform. Sections 4.1–4.2 (p. 7) specify a uniform distribution over classes only for a separate, globally shared dataset used in the authors’ data-sharing strategy. That does not establish that centralized models generally minimize loss with respect to a uniform distribution. (LLM, conf=0.84)
Experimental — accuracy not yet measuredSharing a small subset of data across clients, with uniform class distribution, can address non-IID issues.
“Another commonly used method is globally sharing a small portion of data between all the clients [75, 111]. The shared subset is required containing a uniform distribution over classes from the central server to the clients.”
From Federated Learning for Healthcare Informatics (2021) · cites this paper as [111]
Convolutional neural networks trained with FedAvg can lose substantial performance because of weight divergence.
“The performance of convolutional neural networks trained with FedAvg algorithm can reduce significantly due to the weight divergence [111].”
From Federated Learning for Healthcare Informatics (2021) · cites this paper as [111]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
No citation lists for this paper
No citation lists for this paper yet — checked again daily.
References 24 · Cited by 2,077
Research works printed but not linked: 18
- [1][1] Y. Zhang, N. Suda, L. Lai, and V. Chandra, 'Hello edge: Keyword spotting on microcontrollers,' arXiv preprint arXiv:1711.07128 , 2017. - · arXiv
- [2][2] L. Lai, N. Suda, and V. Chandra, 'Cmsis-nn: Efficient neural network kernels for arm cortex-m cpus,' arXiv preprint arXiv:1801.06601 , 2018. - · arXiv
- [4][4] J. Koneˇ cn` y, B. McMahan, and D. Ramage, 'Federated optimization: Distributed optimization beyond the datacenter,' arXiv preprint arXiv:1511.03575 , 2015. - · arXiv
- [6][6] Y. Lecun, L. Bottou, Y. Bengio, and P. Haffner, 'Gradient-based learning applied to document recognition,' in Proceedings of the IEEE , pp. 2278-2324, 1998. -
- [9][9] K. Bonawitz, V. Ivanov, B. Kreuter, A. Marcedone, H. B. McMahan, S. Patel, D. Ramage, A. Segal, and K. Seth, 'Practical secure aggregation for privacy-preserving machine learning,' in Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security , pp. 1175-1191, ACM, 2017. -
- [10][10] J. Koneˇ cn` y, H. B. McMahan, F. X. Yu, P. Richtárik, A. T. Suresh, and D. Bacon, 'Federated learning: Strategies for improving communication efficiency,' arXiv preprint arXiv:1610.05492 , 2016. - · arXiv
- [11][11] Y. Lin, S. Han, H. Mao, Y. Wang, and W. J. Dally, 'Deep gradient compression: Reducing the communication bandwidth for distributed training,' arXiv preprint arXiv:1712.01887 , 2017. - · arXiv
- [12][12] J. Duchi, E. Hazan, and Y. Singer, 'Adaptive subgradient methods for online learning and stochastic optimization,' Journal of Machine Learning Research , vol. 12, no. Jul, pp. 2121-2159, 2011. -
- [14][14] D. Kinga and J. B. Adam, 'A method for stochastic optimization,' in International Conference on Learning Representations (ICLR) , 2015. -
- [15][15] L. Bottou, 'Large-scale machine learning with stochastic gradient descent,' in Proceedings of COMPSTAT'2010 , pp. 177-186, Springer, 2010. -
- [16][16] A. Rakhlin, O. Shamir, K. Sridharan, et al. , 'Making gradient descent optimal for strongly convex stochastic optimization.,' in ICML , Citeseer, 2012. -
- [17][17] J. Dean, G. Corrado, R. Monga, K. Chen, M. Devin, M. Mao, A. Senior, P. Tucker, K. Yang, Q. V. Le, et al. , 'Large scale distributed deep networks,' in Advances in neural information processing systems , pp. 1223-1231, 2012. -
- [18][18] S. Ghadimi and G. Lan, 'Stochastic first-and zeroth-order methods for nonconvex stochastic programming,' SIAM Journal on Optimization , vol. 23, no. 4, pp. 2341-2368, 2013. -
- [19][19] V. Smith, C.-K. Chiang, M. Sanjabi, and A. S. Talwalkar, 'Federated multi-task learning,' in Advances in Neural Information Processing Systems , pp. 4427-4437, 2017. -
- [20][20] P. Warden, 'Speech commands: A dataset for limited-vocabulary speech recognition,' arXiv preprint arXiv:1804.03209 , 2018. - · arXiv
- [21][21] A. Krizhevsky, I. Sutskever, and G. E. Hinton, 'Imagenet classification with deep convolutional neural networks,' in Advances in neural information processing systems , pp. 1097-1105, 2012. -
- [22][22] B. Graham, 'Fractional max-pooling,' arXiv preprint arXiv:1412.6071 , 2014. - · arXiv
- [23][23] I. J. Goodfellow, O. Vinyals, and A. M. Saxe, 'Qualitatively characterizing neural network optimization problems,' in arXiv preprint arXiv:1412.6544 , 2014. · arXiv
- Authors
- Zhao, Y., Li, M., Lai, L., Suda, N., Civin, D., & Chandra, V.
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2018
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 1806.00582
- OpenAlex
- W2807006176
- Cited by
- 2,077
- References
- 24
- Citations by year
- 2026: 28 · 2025: 213 · 2024: 487 · 2023: 460 · 2022: 326 · 2021: 330 · 2020: 184 · 2019: 46 · 2018: 3
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Privacy-Preserving Technologies in Data
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Physical Sciences › Computer Science › Artificial Intelligence › Stochastic Gradient Optimization Techniques
- Keywords
- federated learning, non-IID data, class skew, Wasserstein distance, neural network accuracy, edge devices, privacy-preserving machine learning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
