CoChat
Sign Up
PreprintCited by 1Privacy-Preserving Technologies in DataAdversarial Robustness in Machine LearningMachine Learning and Data Classification

FADE: Enabling Federated Adversarial Training on Heterogeneous Resource-Constrained Edge Devices

Open access
Tang, Minxue, Zhang, Jianyi, Ma, Mingyuan, DiValentin, Louis, Ding, Aolin, Hassanzadeh, Amin, Li, Hai, Chen, Yiran·2022·arXiv
DOI 10.48550/arxiv.2209.03839arXiv 2209.03839
Open this paper in CoChat — free accountOpen PDF

Abstract

Federated adversarial training can effectively complement adversarial robustness into the privacy-preserving federated learning systems. However, the high demand for memory capacity and computing power makes large-scale federated adversarial training infeasible on resource-constrained edge devices. Few previous studies in federated adversarial training have tried to tackle both memory and computational constraints simultaneously. In this paper, we propose a new framework named Federated Adversarial Decoupled Learning (FADE) to enable AT on heterogeneous resource-constrained edge devices. FADE differentially decouples the entire model into small modules to fit into the resource budget of each device, and each device only needs to perform AT on a single module in each communication round. We also propose an auxiliary weight decay to alleviate objective inconsistency and achieve better accuracy-robustness balance in FADE. FADE offers theoretical guarantees for convergence and adversarial robustness, and our experimental results show that FADE can significantly reduce the consumption of memory and computing power while maintaining accuracy and robustness.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2209.03839

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • The federated adversarial decoupling learning framework significantly reduced computing resources consumed by AT while maintaining almost the same accuracy and robustness as concentrated joint training.

    “The experimental results showed that the federated adversarial decoupling learning framework significantly reduced the computing resources consumed by AT while maintaining almost the same accuracy and robustness as concentrated joint training.”

    From Survey: federated learning data security and privacy-preserving in edge-Internet of Things (2024) · cites this paper as Tang et al. 2022

    The Introduction (p. 2) and Conclusions (p. 8) say FADE significantly reduces computational requirements while maintaining almost the same accuracy and robustness as joint training. But the comparator is joint federated adversarial training: the Experiment Settings (p. 6) identify it as full FedDynAT, not centralized or “concentrated” training. (LLM, conf=0.83)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

No references could be resolved yet.

Research works printed but not linked: 40

  1. Andriushchenko, M. and Flammarion, N. Understanding and improving fast adversarial training. Advances in Neural Information Processing Systems , 33:16048-16059, 2020.
  2. Belilovsky, E., Eickenberg, M., and Oyallon, E. Greedy layerwise learning can scale to imagenet. In International conference on machine learning , pp. 583-593. PMLR, 2019.
  3. Belilovsky, E., Eickenberg, M., and Oyallon, E. Decoupled greedy learning of cnns. In International Conference on Machine Learning , pp. 736-745. PMLR, 2020.
  4. Bhagoji, A. N., Chakraborty, S., Mittal, P., and Calo, S. Analyzing federated learning through an adversarial lens. In International Conference on Machine Learning , pp. 634-643. PMLR, 2019.
  5. Goodfellow, I. J., Shlens, J., and Szegedy, C. Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572 , 2014. · arXiv
  6. Hettinger, C., Christensen, T., Ehlert, B., Humpherys, J., Jarvis, T., and Wade, S. Forward thinking: Building and training neural networks one layer at a time. arXiv preprint arXiv:1706.02480 , 2017. · arXiv
  7. Hong, J., Wang, H., Wang, Z., and Zhou, J. Federated robustness propagation: Sharing adversarial robustness in federated learning. arXiv preprint arXiv:2106.10196 , 2021. · arXiv
  8. Kairouz, P., McMahan, H. B., Avent, B., Bellet, A., Bennis, M., Bhagoji, A. N., Bonawitz, K., Charles, Z., Cormode, G., Cummings, R., et al. Advances and open problems in federated learning. arXiv preprint arXiv:1912.04977 , 2019. · arXiv
  9. Karimireddy, S. P., Kale, S., Mohri, M., Reddi, S. J., Stich, S. U., and Suresh, A. T. Scaffold: Stochastic controlled averaging for on-device federated learning. arXiv preprint arXiv:1910.06378 , 2019. · arXiv
  10. Koneˇ cn` y, J., McMahan, B., and Ramage, D. Federated optimization: Distributed optimization beyond the datacenter. arXiv preprint arXiv:1511.03575 , 2015. · arXiv
  11. Koneˇ cn` y, J., McMahan, H. B., Yu, F. X., Richt´ arik, P., Suresh, A. T., and Bacon, D. Federated learning: Strategies for improving communication efficiency. arXiv preprint arXiv:1610.05492 , 2016. · arXiv
  12. Li, A., Sun, J., Li, P., Pu, Y ., Li, H., and Chen, Y . Hermes: an efficient federated learning framework for heterogeneous mobile clients. In Proceedings of the 27th Annual International Conference on Mobile Computing and Networking , pp. 420-437, 2021a.
  13. Li, T., Sahu, A. K., Zaheer, M., Sanjabi, M., Talwalkar, A., and Smith, V. Federated optimization in heterogeneous networks. arXiv preprint arXiv:1812.06127 , 2018. · arXiv
  14. Li, T., Sahu, A. K., Talwalkar, A., and Smith, V. Federated learning: Challenges, methods, and future directions. IEEE Signal Processing Magazine , 37(3):50-60, 2020.
  15. Li, X., Huang, K., Yang, W., Wang, S., and Zhang, Z. On the convergence of fedavg on non-iid data. arXiv preprint arXiv:1907.02189 , 2019. · arXiv
  16. Li, X., Jiang, M., Zhang, X., Kamp, M., and Dou, Q. Fedbn: Federated learning on non-iid features via local batch normalization. arXiv preprint arXiv:2102.07623 , 2021b. · arXiv
  17. Liang, X., Shen, S., Liu, J., Pan, Z., Chen, E., and Cheng, Y. Variance reduced local sgd with lower communication complexity. arXiv preprint arXiv:1912.12844 , 2019. · arXiv
  18. Liu, X., Cheng, H., He, P., Chen, W., Wang, Y., Poon, H., and Gao, J. Adversarial training for large neural language models. arXiv preprint arXiv:2004.08994 , 2020. · arXiv
  19. Lu, X., Liao, Y., Lio, P., and Hui, P. Privacy-preserving asynchronous federated learning mechanism for edge network computing. IEEE Access , 8:48970-48981, 2020.
  20. Madry, A., Makelov, A., Schmidt, L., Tsipras, D., and Vladu, A. Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv:1706.06083 , 2017. · arXiv
  21. Marquez, E. S., Hare, J. S., and Niranjan, M. Deep cascade learning. IEEE transactions on neural networks and learning systems , 29(11):5475-5485, 2018.
  22. McMahan, B., Moore, E., Ramage, D., Hampson, S., and y Arcas, B. A. Communication-efficient learning of deep networks from decentralized data. In Artificial Intelligence and Statistics , pp. 1273-1282. PMLR, 2017.
  23. Moosavi-Dezfooli, S.-M., Fawzi, A., Uesato, J., and Frossard, P. Robustness via curvature regularization, and vice versa. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pp. 90789086, 2019.
  24. Qin, C., Martens, J., Gowal, S., Krishnan, D., Dvijotham, K., Fawzi, A., De, S., Stanforth, R., and Kohli, P. Adversarial robustness through local linearization. Advances in Neural Information Processing Systems , 32, 2019.
  25. T. Adversarial training for free! Advances in Neural Information Processing Systems , 32, 2019.
  26. Shah, D., Dube, P., Chakraborty, S., and Verma, A. Adversarial training in communication constrained federated learning. arXiv preprint arXiv:2103.01319 , 2021. · arXiv
  27. Shoham, N., Avidor, T., Keren, A., Israel, N., Benditkis, D., Mor-Yosef, L., and Zeitak, I. Overcoming forgetting in federated learning on non-iid data. arXiv preprint arXiv:1910.07796 , 2019. · arXiv
  28. Simonyan, K. and Zisserman, A. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 , 2014. · arXiv
  29. Sun, J., Li, A., DiValentin, L., Hassanzadeh, A., Chen, Y., and Li, H. Fl-wbc: Enhancing robustness against model poisoning attacks in federated learning from a client perspective. Advances in Neural Information Processing Systems , 34, 2021.
  30. Szegedy, C., Zaremba, W., Sutskever, I., Bruna, J., Erhan, D., Goodfellow, I., and Fergus, R. Intriguing properties of neural networks. arXiv preprint arXiv:1312.6199 , 2013. · arXiv
  31. Tang, M., Ning, X., Wang, Y., Sun, J., Wang, Y., Li, H., and Chen, Y. Fedcor: Correlation-based active client selection strategy for heterogeneous federated learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pp. 10102-10111, 2022.
  32. Wang, H., Yurochkin, M., Sun, Y., Papailiopoulos, D., and Khazaeni, Y. Federated learning with matched averaging. arXiv preprint arXiv:2002.06440 , 2020a. · arXiv
  33. Wang, J., Liu, Q., Liang, H., Joshi, G., and Poor, H. V . Tackling the objective inconsistency problem in heterogeneous federated optimization. arXiv preprint arXiv:2007.07481 , 2020b. · arXiv
  34. Wang, Y., Ni, Z., Song, S., Yang, L., and Huang, G. Revisiting locally supervised learning: An alternative to end-to-end training. arXiv preprint arXiv:2101.10832 , 2021. · arXiv
  35. Wong, E., Rice, L., and Kolter, J. Z. Fast is better than free: Revisiting adversarial training. arXiv preprint arXiv:2001.03994 , 2020. · arXiv
  36. Xiao, H., Rasul, K., and Vollgraf, R. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXiv:1708.07747 , 2017. · arXiv
  37. Xie, C., Koyejo, S., and Gupta, I. Asynchronous federated optimization. arXiv preprint arXiv:1903.03934 , 2019. · arXiv
  38. Xie, C., Tan, M., Gong, B., Wang, J., Yuille, A. L., and Le, Q. V. Adversarial examples improve image recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pp. 819-828, 2020.
  39. Zhang, D., Zhang, T., Lu, Y., Zhu, Z., and Dong, B. You only propagate once: Accelerating adversarial training via maximal principle. Advances in Neural Information Processing Systems , 32, 2019.
  40. Zizzo, G., Rawat, A., Sinn, M., and Buesser, B. Fat: Federated adversarial training. arXiv preprint arXiv:2012.01791 , 2020. · arXiv

Cited by

Sorted by citations

Authors
Tang, M., Zhang, J., Ma, M., DiValentin, L., Ding, A., Hassanzadeh, A., Li, H., & Chen, Y.
ORCID
Venue
arXiv
Publisher
arXiv
Published
2022
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2209.03839
arXiv
2209.03839
OpenAlex
W4297726202
Cited by
0
References
0
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Privacy-Preserving Technologies in Data
  • Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Data Classification
Keywords
adversarial robustness, privacy-preserving federated learning, resource-constrained edge devices, model decoupling, heterogeneous edge devices, robustness-accuracy trade-off, convergence guarantees

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review