CoChat
Sign Up
PreprintCited by 250Privacy-Preserving Technologies in DataMachine Learning and Data ClassificationDomain Adaptation and Few-Shot Learning

Astraea: Self-balancing Federated Learning for Improving Classification Accuracy of Mobile Deep Learning Applications

Open access
Duan, Moming, Liu, Duo, Chen, Xianzhang, Tan, Yujuan, Ren, Jinting, Qiao, Lei, Liang, Liang·2019·arXiv
DOI 10.48550/arxiv.1907.01132arXiv 1907.01132
Open this paper in CoChat — free accountOpen PDF

Abstract

Federated learning (FL) is a distributed deep learning method which enables multiple participants, such as mobile phones and IoT devices, to contribute a neural network model while their private training data remains in local devices. This distributed approach is promising in the edge computing system where have a large corpus of decentralized data and require high privacy. However, unlike the common training dataset, the data distribution of the edge computing system is imbalanced which will introduce biases in the model training and cause a decrease in accuracy of federated learning applications. In this paper, we demonstrate that the imbalanced distributed training data will cause accuracy degradation in FL. To counter this problem, we build a self-balancing federated learning framework call Astraea, which alleviates the imbalances by 1) Global data distribution based data augmentation, and 2) Mediator based multi-client rescheduling. The proposed framework relieves global imbalance by runtime data augmentation, and for averaging the local imbalance, it creates the mediator to reschedule the training of clients based on Kullback-Leibler divergence (KLD) of their data distribution. Compared with FedAvg, the state-of-the-art FL algorithm, Astraea shows +5.59% and +5.89% improvement of top-1 accuracy on the imbalanced EMNIST and imbalanced CINIC-10 datasets, respectively. Meanwhile, the communication traffic of Astraea can be 82% lower than that of FedAvg.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.1907.01132

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • [212] improves Top 1 accuracy by 5.59%-5.89% and reduces communication traffic by 92%.

    “Top 1 accuracy imrpovement: 5.59%-5.89%;Communication traffic reduction: 92%”

    From Convergence of Edge Computing and Deep Learning: A Comprehensive Survey (2020) · cites this paper as [212]

    The Conclusion (p. 8) reports top-1 accuracy gains of 5.59% on imbalanced EMNIST and 5.89% on imbalanced CINIC-10 versus FedAvg. The Overhead section (p. 8), however, reports an 81.7% reduction in communication cost for its measured comparison, not 92%. (LLM, conf=0.91)

    Experimental — accuracy not yet measured

  • Astraea reduces 92% communication traffic by designing a mediator-based multi-client rescheduling strategy.

    “Astraea [212] reduces 92% communication traffic by designing a mediator-based multi-client rescheduling strategy.”

    From Convergence of Edge Computing and Deep Learning: A Comprehensive Survey (2020) · cites this paper as [212]

    The Introduction (p. 1) reports a 92% reduction in communication traffic relative to FedAvg when reaching 75% accuracy on imbalanced EMNIST, and describes mediator-based rescheduling. The citing sentence omits those conditions and attributes the reduction to rescheduling alone, although the Introduction describes data augmentation as part of Astraea too. The detailed Overhead results (p. 8) instead calculate an 81.7% reduction for reaching 75% accuracy. (LLM, conf=0.88)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

No references could be resolved yet.

Research works printed but not linked: 30

  1. [1][1] A. Krizhevsky, I. Sutskever, and G. E. Hinton, 'Imagenet classification with deep convolutional neural networks,' in Advances in neural information processing systems , 2012, pp. 1097-1105. -
  2. [2][2] G. Hinton, L. Deng, D. Yu, G. Dahl, A.-r. Mohamed, N. Jaitly, A. Senior, V. Vanhoucke, P. Nguyen, B. Kingsbury et al. , 'Deep neural networks for acoustic modeling in speech recognition,' IEEE Signal processing magazine , vol. 29, 2012. -
  3. [3][3] H. B. McMahan, E. Moore, D. Ramage, S. Hampson et al. , 'Communication-efficient learning of deep networks from decentralized data,' arXiv preprint arXiv:1602.05629 , 2016. - · arXiv
  4. [4][4] Y. Zhao, M. Li, L. Lai, N. Suda, D. Civin, and V. Chandra, 'Federated learning with non-iid data,' arXiv preprint arXiv:1806.00582 , 2018. - · arXiv
  5. [5][5] S. C. Wong, A. Gatt, V. Stamatescu, and M. D. McDonnell, 'Understanding data augmentation for classification: when to warp?' in 2016 international conference on digital image computing: techniques and applications (DICTA) . IEEE, 2016, pp. 1-6. -
  6. [6][6] L. N. Darlow, E. J. Crowley, A. Antoniou, and A. J. Storkey, 'Cinic-10 is not imagenet or cifar-10,' arXiv preprint arXiv:1810.03505 , 2018. - · arXiv
  7. [7][7] M. Abadi, P. Barham, J. Chen, Z. Chen, A. Davis, J. Dean, M. Devin, S. Ghemawat, G. Irving, M. Isard et al. , 'Tensorflow: A system for largescale machine learning,' in 12th { USENIX } Symposium on Operating Systems Design and Implementation ( { OSDI } 16) , 2016, pp. 265-283. -
  8. [8][8] A. Hard, K. Rao, R. Mathews, F. Beaufays, S. Augenstein, H. Eichner, C. Kiddon, and D. Ramage, 'Federated learning for mobile keyboard prediction,' arXiv preprint arXiv:1811.03604 , 2018. - · arXiv
  9. [9][9] K. Bonawitz, H. Eichner, W. Grieskamp, D. Huba, A. Ingerman, V. Ivanov, C. Kiddon, J. Konecny, S. Mazzocchi, H. B. McMahan et al. , 'Towards federated learning at scale: System design,' in Proceedings of the 2nd SysML Conference , 2019. -
  10. [10][10] J. Koneˇ cn` y, H. B. McMahan, F. X. Yu, P. Richt´ arik, A. T. Suresh, and D. Bacon, 'Federated learning: Strategies for improving communication efficiency,' arXiv preprint arXiv:1610.05492 , 2016. - · arXiv
  11. [11][11] S. Samarakoon, M. Bennis, W. Saad, and M. Debbah, 'Federated learning for ultra-reliable low-latency v2v communications,' in 2018 IEEE Global Communications Conference (GLOBECOM) . IEEE, 2018, pp. 1-7. -
  12. [12][12] D. Liu, C. Yang, S. Li, X. Chen, J. Ren, R. Liu, M. Duan, Y. Tan, and L. Liang, 'Fitcnn: A cloud-assisted and low-cost framework for updating cnns on iot devices,' Future Generation Computer Systems , vol. 91, pp. 277-289, 2019. -
  13. [13][13] K. Bonawitz, V. Ivanov, B. Kreuter, A. Marcedone, H. B. McMahan, S. Patel, D. Ramage, A. Segal, and K. Seth, 'Practical secure aggregation for privacy-preserving machine learning,' in Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security . ACM, 2017, pp. 1175-1191. -
  14. [14][14] N. Agarwal, A. T. Suresh, F. X. X. Yu, S. Kumar, and B. McMahan, 'cpsgd: Communication-efficient and differentially-private distributed sgd,' in Advances in Neural Information Processing Systems , 2018, pp. 7564-7575. -
  15. [15][15] Z. Wang, M. Song, Z. Zhang, Y. Song, Q. Wang, and H. Qi, 'Beyond inferring class representatives: User-level privacy leakage from federated learning,' in IEEE INFOCOM 2019-IEEE Conference on Computer Communications . IEEE, 2019, pp. 2512-2520. -
  16. [16][16] G. Xu, H. Li, S. Liu, K. Yang, and X. Lin, 'Verifynet: Secure and verifiable federated learning,' IEEE Transactions on Information Forensics and Security , 2019. -
  17. [17][17] F. Sattler, S. Wiedemann, K.-R. M¨ uller, and W. Samek, 'Robust and communication-efficient federated learning from non-iid data,' arXiv preprint arXiv:1903.02891 , 2019. - · arXiv
  18. [18][18] H. He and E. A. Garcia, 'Learning from imbalanced data,' IEEE Transactions on Knowledge & Data Engineering , no. 9, pp. 1263-1284, 2008. -
  19. [19][19] N. V. Chawla, K. W. Bowyer, L. O. Hall, and W. P. Kegelmeyer, 'Smote: synthetic minority over-sampling technique,' Journal of artificial intelligence research , vol. 16, pp. 321-357, 2002. -
  20. [20][20] H. Han, W.-Y. Wang, and B.-H. Mao, 'Borderline-smote: a new oversampling method in imbalanced data sets learning,' in International conference on intelligent computing . Springer, 2005, pp. 878-887. -
  21. [21][21] G. R¨ atsch, T. Onoda, and K.-R. M¨ uller, 'Soft margins for adaboost,' Machine learning , vol. 42, no. 3, pp. 287-320, 2001. -
  22. [22][22] T. Chen and C. Guestrin, 'Xgboost: A scalable tree boosting system,' in Proceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining . ACM, 2016, pp. 785-794. -
  23. [23][23] L. A. Adamic and B. A. Huberman, 'Zipf's law and the internet.' Glottometrics , vol. 3, no. 1, pp. 143-150, 2002. -
  24. [24][24] G. Cohen, S. Afshar, J. Tapson, and A. van Schaik, 'Emnist: Extending mnist to handwritten letters,' in 2017 International Joint Conference on Neural Networks (IJCNN) . IEEE, 2017, pp. 2921-2926. -
  25. [25][25] S. Caldas, P. Wu, T. Li, J. Koneˇ cn` y, H. B. McMahan, V. Smith, and A. Talwalkar, 'Leaf: A benchmark for federated settings,' arXiv preprint arXiv:1812.01097 , 2018. - · arXiv
  26. [27][27] L. Wan, M. Zeiler, S. Zhang, Y. Le Cun, and R. Fergus, 'Regularization of neural networks using dropconnect,' in International conference on machine learning , 2013, pp. 1058-1066. -
  27. [28][28] D. P. Kingma and J. Ba, 'Adam: A method for stochastic optimization,' arXiv preprint arXiv:1412.6980 , 2014. - · arXiv
  28. [29][29] Y. LeCun, Y. Bengio, and G. Hinton, 'Deep learning,' nature , vol. 521, no. 7553, p. 436, 2015. -
  29. [31][31] T. Yang, G. Andrew, H. Eichner, H. Sun, W. Li, N. Kong, D. Ramage, and F. Beaufays, 'Applied federated learning: Improving google keyboard query suggestions,' arXiv preprint arXiv:1812.02903 , 2018. - · arXiv
  30. [32][32] Y. Yao, L. Rosasco, and A. Caponnetto, 'On early stopping in gradient descent learning,' Constructive Approximation , vol. 26, no. 2, pp. 289315, 2007.

Cited by

Sorted by citations

Authors
Duan, M., Liu, D., Chen, X., Tan, Y., Ren, J., Qiao, L., & Liang, L.
Affiliations
  • Chongqing University (CN)
  • Beijing Institute of Control Engineering (CN)
Venue
arXiv
Publisher
arXiv
Published
2019
Pages
246-254
ISSN
1063-6404 (ISSN-L)
DOI
10.48550/arxiv.1907.01132
arXiv
1907.01132
OpenAlex
W2954124071
Cited by
250
References
0
Citations by year
2026: 14 · 2025: 32 · 2024: 65 · 2023: 56 · 2022: 40 · 2021: 29 · 2020: 13 · 2019: 1
FWCI
7.94
Type
conference paper
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • IEEE International Conference on Computer Design: VLSI in Computers and Processors (ICCD) (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Privacy-Preserving Technologies in Data
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Data Classification
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
Keywords
federated learning, class imbalance, data augmentation, Kullback-Leibler divergence, edge computing, mobile deep learning, classification accuracy, communication efficiency, EMNIST

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review