The DOI matches its Crossref record; 62 of the 62 references it cites were found in a registry or an index.
Federated Learning on Non-IID Data Silos: An Experimental Study
Open accessAbstract
Due to the increasing privacy concerns and data regulations, training data have been increasingly fragmented, forming distributed databases of multiple “data silos” (e.g., within different organizations and countries). To develop effective machine learning services, there is a must to exploit data from such distributed databases without exchanging the raw data. Recently, federated learning (FL) has been a solution with growing interests, which enables multiple parties to collaboratively train a machine learning model without exchanging their local data. A key and common challenge on distributed databases is the heterogeneity of the data distribution among the parties. The data of different parties are usually non-independently and identically distributed (i.e., non-IID). There have been many FL algorithms to address the learning effectiveness under non-IID data settings. However, there lacks an experimental study on systematically understanding their advantages and disadvantages, as previous studies have very rigid data partitioning strategies among parties, which are hardly representative and thorough. In this paper, to help researchers better understand and study the non-IID data setting in federated learning, we propose comprehensive data partitioning strategies to cover the typical non-IID data cases. Moreover, we conduct extensive experiments to evaluate state-of-the-art FL algorithms. We find that non-IID does bring significant challenges in learning accuracy of FL algorithms, and none of the existing state-of-the-art FL algorithms outperforms others in all cases. Our experiments provide insights for future studies of addressing the challenges in “data silos”.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Gradient-based learning applied to document recognition
Lecun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998) · Proceedings of the IEEE · doi:10.1109/5.726791 · Cited by 58,805
(2008) · Journal of Machine Learning Research · Cited by 33,051
Learning Multiple Layers of Features from Tiny Images
Krizhevsky, A. (2024) · doi:10.57702/zp44cu3g · Cited by 23,395
Beyond a Gaussian Denoiser: Residual Learning of Deep CNN for Image Denoising
Zhang, K., Zuo, W., Chen, Y., Meng, D., & Zhang, L. (2017) · IEEE Transactions on Image Processing · doi:10.1109/tip.2017.2662206 · Cited by 8,983
Federated Machine Learning: Concept and Applications
Yang, Q., Liu, Y., Chen, T., & Tong, Y. (2019) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3298981 · Cited by 6,322
Federated learning: Challenges, methods, and future directions
Li, T., Sahu, A. K., Talwalkar, A., & Smith, V. (2020) · IEEE Signal Processing Magazine · doi:10.1109/msp.2020.2975749 · Cited by 6,210
Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
Xiao, H., Rasul, K., & Vollgraf, R. (2017) · arXiv · doi:10.48550/arxiv.1708.07747 · Cited by 6,108
Membership Inference Attacks Against Machine Learning Models
Shokri, R., Stronati, M., Song, C., & Shmatikov, V. (2017) · 2017 IEEE Symposium on Security and Privacy (SP) · doi:10.1109/sp.2017.41 · Cited by 4,754
Reading digits in natural images with unsupervised feature learning
Netzer, Y. (2024) · doi:10.57702/e7wwo0zk · Cited by 4,149
Federated Optimization in Heterogeneous Networks
Li, T., Sahu, A. K., Zaheer, M., Sanjabi, M., Talwalkar, A., & Smith, V. (2018) · arXiv · doi:10.48550/arxiv.1812.06127 · Cited by 4,023
The EU General Data Protection Regulation (GDPR)
Voigt, P., & von dem Bussche, A. (2017) · doi:10.1007/978-3-319-57959-7 · Cited by 3,001
Model Inversion Attacks that Exploit Confidence Information and Basic Countermeasures
Fredrikson, M., Jha, S., & Ristenpart, T. (2015) · Proceedings of the 22nd ACM SIGSAC Conference on Computer and Communications Security · doi:10.1145/2810103.2813677 · Cited by 2,961
Federated database systems for managing distributed, heterogeneous, and autonomous databases
Sheth, A. P., & Larson, J. A. (1990) · ACM Computing Surveys · doi:10.1145/96602.96604 · Cited by 2,476
Accelerating Stochastic Gradient Descent using Predictive Variance Reduction
Johnson, R., & Zhang, T. (2013) · Cited by 1,889
Privacy-preserving data mining
Agrawal, R., & Srikant, R. (2000) · Proceedings of the 2000 ACM SIGMOD international conference on Management of data · doi:10.1145/342009.335438 · Cited by 1,719
Secure, privacy-preserving and federated machine learning in medical imaging
Kaissis, G. A., Makowski, M. R., Rückert, D., & Braren, R. F. (2020) · Nature Machine Intelligence · doi:10.1038/s42256-020-0186-1 · Cited by 1,676 · cited 1×
“It has attracted many research interests [7, 22, 31, 36, 39, 45, 64] and been widely used in practice [3, 20, 30].”
BackgroundEMNIST: Extending MNIST to handwritten letters
Cohen, G., Afshar, S., Tapson, J., & van Schaik, A. (2017) · 2017 International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.2017.7966217 · Cited by 1,651
On the Convergence of FedAvg on Non-IID Data
Li, X., Huang, K., Yang, W., Wang, S., & Zhang, Z. (2019) · arXiv · doi:10.48550/arxiv.1907.02189 · Cited by 1,575
Model-Contrastive Federated Learning
Li, Q., He, B., & Song, D. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.01057 · Cited by 1,534
Self-citationA Survey on Federated Learning Systems: Vision, Hype and Reality for Data Privacy and Protection
Li, Q., Wen, Z., Wu, Z., Hu, S., Wang, N., Li, Y., Liu, X., & He, B. (2023) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2021.3124599 · Cited by 1,407
Self-citationFederated Learning for Mobile Keyboard Prediction
Hard, A., Rao, K., Mathews, R., Ramaswamy, S., Beaufays, F., Augenstein, S., Eichner, H., Kiddon, C., & Ramage, D. (2018) · arXiv · doi:10.48550/arxiv.1811.03604 · Cited by 1,148
Neyman, J. (1934) · Journal of the Royal Statistical Society · doi:10.2307/2342192 · Cited by 1,086
Neyman, J. (1992) · Springer Series in Statistics · doi:10.1007/978-1-4612-4380-9_12 · Cited by 932
Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimization
Wang, J., Liu, Q., Liang, H., Joshi, G., & Poor, H. V. (2020) · arXiv · doi:10.48550/arxiv.2007.07481 · Cited by 927
Minimizing finite sums with the stochastic average gradient
Schmidt, M., Le Roux, N., & Bach, F. (2017) · Mathematical Programming · doi:10.1007/s10107-016-1030-6 · Cited by 922
Adaptive Federated Optimization
Reddi, S., Charles, Z., Zaheer, M., Garrett, Z., Rush, K., Konečný, J., Kumar, S., & McMahan, H. B. (2020) · arXiv · doi:10.48550/arxiv.2003.00295 · Cited by 840
A secure federated transfer learning framework
Liu, Y., Kang, Y., Xing, C., Chen, T., & Yang, Q. (2020) · IEEE Intelligent Systems · doi:10.1109/mis.2020.2988525 · Cited by 645
Secure Federated Transfer Learning
Liu, Y., Kang, Y., Xing, C., Chen, T., & Yang, Q. (2018) · arXiv · doi:10.48550/arxiv.1812.03337 · Cited by 645
Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classification
Hsu, T.-M. H., Qi, H., & Brown, M. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.06335 · Cited by 610
Personalized Federated Learning with Theoretical Guarantees: A Model-Agnostic Meta-Learning Approach
Fallah, A., Mokhtari, A., & Ozdaglar, A. (2020) · neural information processing systems · Cited by 596
Maintaining Data Privacy in Association Rule Mining
Rizvi, S. J., & Haritsa, J. R. (2002) · VLDB '02: Proceedings of the 28th International Conference on Very Large Databases · doi:10.1016/b978-155860869-6/50066-4 · Cited by 485
Ensemble Distillation for Robust Model Fusion in Federated Learning
Lin, T., Kong, L., Stich, S. U., & Jaggi, M. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.07242 · Cited by 476
Neo: a learned query optimizer
Marcus, R., Negi, P., Mao, H., Zhang, C., Alizadeh, M., Kraska, T., Papaemmanouil, O., & Tatbul, N. (2019) · Proceedings of the VLDB Endowment · doi:10.14778/3342263.3342644 · Cited by 389
Mohri, M., Sivek, G., & Suresh, A. T. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1902.00146 · Cited by 373
Dwork, C. (2011) · Encyclopedia of Cryptography and Security · doi:10.1007/978-1-4419-5906-5_752 · Cited by 339
FedBN: Federated Learning on Non-IID Features via Local Batch Normalization
Li, X., Jiang, M., Zhang, X., Kamp, M., & Dou, Q. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2102.07623 · Cited by 332
FedMCCS: Multicriteria Client Selection Model for Optimal IoT Federated Learning
Abdulrahman, S., Tout, H., Mourad, A., & Talhi, C. (2021) · IEEE Internet of Things Journal · doi:10.1109/jiot.2020.3028742 · Cited by 320
FedML: A Research Library and Benchmark for Federated Machine Learning
He, C., Li, S., So, J., Zeng, X., Zhang, M., Wang, H., Wang, X., Vepakomma, P., Singh, A., Qiu, H., et al. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2007.13518 · Cited by 314
Federated Adversarial Domain AdaptationPreprint
Peng, X., Huang, Z., Zhu, Y., & Saenko, K. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1911.02054 · Cited by 294
One-Shot Federated LearningPreprint
Guha, N., Talwalkar, A., & Smith, V. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1902.11175 · Cited by 285
Addressing Class Imbalance in Federated Learning
Wang, L., Xu, S., Wang, X., & Zhu, Q. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i11.17219 · Cited by 251
Fast, small-space algorithms for approximate histogram maintenance
Gilbert, A. C., Guha, S., Indyk, P., Kotidis, Y., Muthukrishnan, S., & Strauss, M. J. (2002) · Proceedings of the thiry-fourth annual ACM symposium on Theory of computing · doi:10.1145/509907.509966 · Cited by 242
SCAFFOLD: Stochastic Controlled Averaging for On-Device Federated Learning.
Karimireddy, S. P., Kale, S., Mohri, M., Reddi, S. J., Stich, S. U., & Suresh, A. T. (2019) · Cited by 227
Practical Federated Gradient Boosting Decision Trees
Li, Q., Wen, Z., & He, B. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i04.5895 · Cited by 201
Self-citationPrivacy preserving vertical federated learning for tree-based models
Wu, Y., Cai, S., Xiao, X., Chen, G., & Ooi, B. C. (2020) · Proceedings of the VLDB Endowment · doi:10.14778/3407790.3407811 · Cited by 184
Group Knowledge Transfer: Federated Learning of Large CNNs at the Edge
He, C., Annavaram, M., & Avestimehr, S. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2007.14513 · Cited by 177
Siloed Federated Learning for Multi-centric Histopathology Datasets
Andreux, M., du Terrail, J. O., Béguier, C., & Tramel, E. W. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-60548-3_13 · Cited by 164
Practical One-Shot Federated Learning for Cross-Silo Setting
Li, Q., He, B., & Song, D. X. (2021) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2021/205 · Cited by 151
Self-citationRobust Federated Learning: The Case of Affine Distribution ShiftsPreprint
Reisizadeh, A., Farnia, F., Pedarsani, R., & Jadbabaie, A. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.08907 · Cited by 113
Lower Bounds and Optimal Algorithms for Personalized Federated LearningPreprint
Hanzely, F., Hanzely, S., Horváth, S., & Richtárik, P. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.02372 · Cited by 109
Rafiki: machine learning as an analytics service system
Wang, W., Gao, J., Zhang, M., Wang, S., Chen, G., Ng, T. K., Ooi, B. C., Shao, J., & Reyad, M. (2018) · Proceedings of the VLDB Endowment · doi:10.14778/3282495.3282499 · Cited by 102
Federated Learning with Only Positive LabelsPreprint
Yu, F. X., Rawat, A. S., Menon, A. K., & Kumar, S. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2004.10342 · Cited by 102
A demonstration of sterling: a privacy-preserving data marketplace
Hynes, N., Dao, D., Yan, D., Cheng, R., & Song, D. X. (2018) · Proceedings of the VLDB Endowment · doi:10.14778/3229863.3236266 · Cited by 88
Federated Bayesian Optimization via Thompson SamplingPreprint
Dai, Z., Low, B. K. H., & Jaillet, P. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.10154 · Cited by 77
The OARF Benchmark Suite: Characterization and Implications for Federated Learning Systems
Hu, S., Li, Y., Liu, X., Li, Q., Wu, Z., & He, B. (2022) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3510540 · Cited by 73
Self-citationUnderstanding and benchmarking the impact of GDPR on database systems
Shastri, S., Banakar, V., Wasserman, M., Kumar, A., & Chidambaram, V., V (2020) · Proceedings of the VLDB Endowment · doi:10.14778/3384345.3384354 · Cited by 71
The Machine Learning Bazaar: Harnessing the ML Ecosystem for Effective System Development
Smith, M. J., Sala, C., Kanter, J. M., & Veeramachaneni, K. K. (2020) · ACM SIGMOD International Conference on Management of Data (SIGMOD) · doi:10.1145/3318464.3386146 · Cited by 55
The Machine Learning Bazaar: Harnessing the ML Ecosystem for Effective System Development
Smith, M. J., Sala, C., Kanter, J. M., & Veeramachaneni, K. K. (2020) · DSpace@MIT (Massachusetts Institute of Technology) · Cited by 46
Trading Data in Good Faith: Integrating Truthfulness and Privacy Preservation in Data Markets
Niu, C., Zheng, Z., Wu, F., et al. (2017) · Proceedings - International Conference on Data Engineering · doi:10.1109/icde.2017.80 · Cited by 45
Ease.ml: Towards Multi-tenant Resource Sharing for Machine Learning WorkloadsPreprint
Li, T., Zhong, J., Liu, J., Wu, W., & Zhang, C. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1708.07308 · Cited by 33
Evaluation Framework For Large-scale Federated LearningPreprint
Liu, L., Zhang, F., Xiao, J., & Wu, C. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2003.01575 · Cited by 15
An Isolated Data Island Benchmark Suite for Federated Learning.Preprint
Liang, Y., Guo, Y., Gong, Y., et al. (2020) · arXiv (Cornell University) · Cited by 4
Cited by
Model-Contrastive Federated Learning
Li, Q., He, B., & Song, D. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.01057 · Cited by 1,534
Self-citationA Survey on Federated Learning Systems: Vision, Hype and Reality for Data Privacy and Protection
Li, Q., Wen, Z., Wu, Z., Hu, S., Wang, N., Li, Y., Liu, X., & He, B. (2023) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2021.3124599 · Cited by 1,407
Self-citationFederated learning on non-IID data: A survey
Zhu, H., Xu, J., Liu, S., & Jin, Y. (2021) · Neurocomputing · doi:10.1016/j.neucom.2021.07.098 · Cited by 1,062
Federated learning for smart healthcare: A survey
Nguyen, D. C., Pham, Q.-V., Pathirana, P. N., Ding, M., Seneviratne, A., Lin, Z., Dobre, O., & Hwang, W.-J. (2023) · ACM Computing Surveys · doi:10.1145/3501296 · Cited by 827
Federated Learning With Non-IID Data: A Survey
Lu, Z., Pan, H., Dai, Y., Si, X., & Zhang, Y. (2024) · IEEE Internet of Things Journal · doi:10.1109/jiot.2024.3376548 · Cited by 430
Metadata
- Authors
- Li, Q., Diao, Y., Chen, Q., & He, B.
- Venue
- 2022 IEEE 38th International Conference on Data Engineering (ICDE)
- Publisher
- IEEE
- Published
- 2022-05
- Pages
- 965-978
- ISSN
- 1084-4627 (ISSN-L)
- OpenAlex
- W4287332481
- Cited by
- 1,132
- References
- 62
- FWCI
- 140.49
- Type
- conference paper
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.