CoChat
Sign Up
PreprintCited by 131Privacy-Preserving Technologies in DataIoT and Edge/Fog ComputingStochastic Gradient Optimization Techniques

Multi-Stage Hybrid Federated Learning over Large-Scale D2D-Enabled Fog Networks

Open access
Hosseinalipour, Seyyedali, Azam, Sheikh Shams, Brinton, Christopher G., Michelusi, Nicolo, Aggarwal, Vaneet, Love, David J., Dai, Huaiyu·2020·arXiv
DOI 10.48550/arxiv.2007.09511arXiv 2007.09511
Open this paper in CoChat — free accountOpen PDF

Abstract

Federated learning has generated significant interest, with nearly all works focused on a "star" topology where nodes/devices are each connected to a central server. We migrate away from this architecture and extend it through the network dimension to the case where there are multiple layers of nodes between the end devices and the server. Specifically, we develop multi-stage hybrid federated learning (MH-FL), a hybrid of intra- and inter-layer model learning that considers the network as a multi-layer cluster-based structure. MH-FL considers the topology structures among the nodes in the clusters, including local networks formed via device-to-device (D2D) communications, and presumes a semi-decentralized architecture for federated learning. It orchestrates the devices at different network layers in a collaborative/cooperative manner (i.e., using D2D interactions) to form local consensus on the model parameters and combines it with multi-stage parameter relaying between layers of the tree-shaped hierarchy. We derive the upper bound of convergence for MH-FL with respect to parameters of the network topology (e.g., the spectral radius) and the learning algorithm (e.g., the number of D2D rounds in different clusters). We obtain a set of policies for the D2D rounds at different clusters to guarantee either a finite optimality gap or convergence to the global optimum. We then develop a distributed control algorithm for MH-FL to tune the D2D rounds in each cluster over time to meet specific convergence criteria. Our experiments on real-world datasets verify our analytical results and demonstrate the advantages of MH-FL in terms of resource utilization metrics.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2007.09511

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Distributed inference approaches can keep the source data and perform pre-processing on the source trusted machine, then perform neural network processing on a normal potentially untrusted machine.

    “For such applications, approaches based on distributed inference [136] can be achieved that keep the source data (e.g., captured images) and perform the pre-processing step on the source (trusted) machine, and then the neural network processing can be performed on a normal (potentially untrusted) machine [76].”

    From Confidential Computing Across Edge‐To‐Cloud for Machine Learning: A Survey Study (2025) · cites this paper as [136]

    The Introduction (p. 1) describes keeping data on devices during federated model training, not performing neural-network inference on a potentially untrusted machine. The Machine Learning Task section (p. 4) likewise concerns local training and model aggregation. Searches for distributed inference, preprocessing, and trusted or untrusted machines found no passage addressing the part of the sentence attributed to this source. (LLM, conf=0.75)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

No references could be resolved yet.

Research works printed but not linked: 54

  1. [1][1] D. Ciregan, U. Meier, and J. Schmidhuber, 'Multi-column deep neural networks for image classification,' in Proc. IEEE Conf. Comput. Vision Pattern Recog. (CVPR) , 2012, pp. 3642-3649. -
  2. [2][2] R. Collobert and J. Weston, 'A unified architecture for natural language processing: Deep neural networks with multitask learning,' in Proc. Int. Conf. Mach. Learn. , 2008, pp. 160-167. -
  3. [3][3] B. McMahan, E. Moore, D. Ramage, S. Hampson, and B. A. y Arcas, 'Communication-efficient learning of deep networks from decentralized data,' in Proc. Artif. Intell. Stat. , 2017, pp. 1273-1282. -
  4. [4][4] S. Hosseinalipour, C. G. Brinton, V. Aggarwal, H. Dai, and M. Chiang, 'From federated to fog learning: Distributed machine learning over heterogeneous wireless networks,' IEEE Commun. Mag. , vol. 58, no. 12, pp. 41-47, 2020. -
  5. [5][5] M. Chiang, S. Ha, F. Risso, T. Zhang, and I. Chih-Lin, 'Clarifying fog computing and networking: 10 questions and answers,' IEEE Commun. Mag. , vol. 55, no. 4, pp. 18-20, 2017. -
  6. [6][6] M. N. Tehrani, M. Uysal, and H. Yanikomeroglu, 'Device-to-device communication in 5G cellular networks: challenges, solutions, and future directions,' IEEE Commun. Mag. , vol. 52, no. 5, pp. 86-92, 2014. -
  7. [7][7] M. Abolhasan, T. Wysocki, and E. Dutkiewicz, 'A review of routing protocols for mobile ad hoc networks,' Ad hoc Netw. , vol. 2, no. 1, pp. 1-22, 2004. -
  8. [8][8] S. Zeadally, R. Hunt, Y.-S. Chen, A. Irwin, and A. Hassan, 'Vehicular ad hoc networks (VANETS): status, results, and challenges,' Telecommun. Syst. , vol. 50, no. 4, pp. 217-241, 2012. -
  9. [9][9] I. Bekmezci, O. K. Sahingoz, and S ¸. Temel, 'Flying ad-hoc networks (FANETs): A survey,' Ad hoc Netw. , vol. 11, no. 3, pp. 1254-1270, 2013. -
  10. [10][10] K. Akkaya and M. Younis, 'A survey on routing protocols for wireless sensor networks,' Ad hoc Netw. , vol. 3, no. 3, pp. 325-349, 2005. -
  11. [11][11] Y. Zhang, D. J. Love, J. V. Krogmeier, C. R. Anderson, R. W. Heath, and D. R. Buckmaster, 'Challenges and opportunities of future rural wireless communications,' IEEE Commun. Mag. , 2021. -
  12. [12][12] S. Maheshwari, D. Raychaudhuri, I. Seskar, and F. Bronzino, 'Scalability and performance evaluation of edge cloud systems for latency constrained applications,' in IEEE/ACM Symp. Edge Comput. , 2018, pp. 286-299. -
  13. [13][13] M. Chen, Z. Yang, W. Saad, C. Yin, H. V. Poor, and S. Cui, 'A joint learning and communications framework for federated learning over wireless networks,' arXiv preprint arXiv:1909.07972 , 2019. - · arXiv
  14. [14][14] N. H. Tran, W. Bao, A. Zomaya, M. N. H. Nguyen, and C. S. Hong, 'Federated learning over wireless networks: Optimization model design and analysis,' in Proc. IEEE Int. Conf. Comput. Commun. (INFOCOM) , 2019, pp. 1387-1395. -
  15. [15][15] M. M. Amiri and D. G¨ und¨ uz, 'Federated learning over wireless fading channels,' IEEE Trans. Wireless Commun. , vol. 19, no. 5, pp. 3546-3557, 2020. -
  16. [16][16] N. Shlezinger, M. Chen, Y. C. Eldar, H. V. Poor, and S. Cui, 'Federated learning with quantization constraints,' in Proc. IEEE Int. Conf. Acoustics, Speech, Signal Process. (ICASSP) , 2020, pp. 8851-8855. -
  17. [17][17] C.-S. Lee, N. Michelusi, and G. Scutari, 'Finite-bit quantization for distributed algorithms with linear convergence,' arXiv preprint arXiv:2107.11304 , 2021. - · arXiv
  18. [18][18] C. Renggli, S. Ashkboos, M. Aghagolzadeh, D. Alistarh, and T. Hoefler, 'SparCML: High-performance sparse communication for machine learning,' in Proc. Int. Conf. High Perform. Comput., Netw., Storage Anal. , 2019, pp. 1-15. -
  19. [19][19] S. Wang, T. Tuor, T. Salonidis, K. K. Leung, C. Makaya, T. He, and K. Chan, 'Adaptive federated learning in resource constrained edge computing systems,' IEEE J. Sel. Areas Commun. (JSAC) , vol. 37, no. 6, pp. 1205-1221, 2019. -
  20. [20][20] D. Ye, R. Yu, M. Pan, and Z. Han, 'Federated learning in vehicular edge computing: A selective model aggregation approach,' IEEE Access , vol. 8, pp. 23 920-23 935, 2020. -
  21. [21][21] Y. Tu, Y. Ruan, S. Wang, S. Wagle, C. G. Brinton, and C. Joe-Wang, 'Network-aware optimization of distributed learning for fog computing,' in Proc. IEEE Int. Conf. Comput. Commun. (INFOCOM) , 2020. -
  22. [22][22] S. Wang, M. Lee, S. Hosseinalipour, R. Morabito, M. Chiang, and C. G. Brinton, 'Device sampling for heterogeneous federated learning: Theory, algorithms, and implementation,' in IEEE Conf. Comput. Commun. (INFOCOM) , 2021, pp. 1-10. -
  23. [23][23] S. Ji, W. Jiang, A. Walid, and X. Li, 'Dynamic sampling and selective masking for communication-efficient federated learning,' arXiv preprint arXiv:2003.09603 , 2020. - · arXiv
  24. [24][24] W. Wang, Y. Sun, B. Eriksson, W. Wang, and V. Aggarwal, 'Wide compression: Tensor ring nets,' in Proc. IEEE Conf. Comput. Vision Pattern Recog. (CVPR) , 2018, pp. 9329-9338. -
  25. [25][25] L. Liu, J. Zhang, S. Song, and K. B. Letaief, 'Client-edge-cloud hierarchical federated learning,' in Proc. IEEE Int. Conf. Commun. (ICC) , 2020, pp. 1-6. -
  26. [26][26] S. Luo, X. Chen, Q. Wu, Z. Zhou, and S. Yu, 'HFEL: Joint edge association and resource allocation for cost-efficient hierarchical federated edge learning,' IEEE Trans. Wireless Commun. , pp. 1-1, 2020. -
  27. [27][27] M. S. H. Abad, E. Ozfatura, D. GUndUz, and O. Ercetin, 'Hierarchical federated learning across heterogeneous cellular networks,' in Proc. IEEE ICASSP , 2020, pp. 8866-8870. -
  28. [28][28] S. Savazzi, M. Nicoli, and V. Rampa, 'Federated learning with cooperating devices: A consensus approach for massive IoT networks,' IEEE Internet Things J. , vol. 7, no. 5, pp. 4641-4654, 2020. -
  29. [29][29] C. Hu, J. Jiang, and Z. Wang, 'Decentralized federated learning: A segmented gossip approach,' arXiv preprint arXiv:1908.07782 , 2019. - · arXiv
  30. [30][30] A. Elgabli, J. Park, A. S. Bedi, M. Bennis, and V. Aggarwal, 'GADMM: Fast and communication efficient framework for distributed machine learning,' arXiv preprint arXiv:1909.00047 , 2019. - · arXiv
  31. [31][31] V. Smith, S. Forte, C. Ma, M. Tak´ aˇ c, M. I. Jordan, and M. Jaggi, 'CoCoA: A general framework for communication-efficient distributed optimization,' J. Mach. Learn. Res. , vol. 18, no. 1, pp. 8590-8638, 2017. -
  32. [32][32] P. Richt´ arik and M. Tak´ aˇ c, 'Distributed coordinate descent method for learning with big data,' J. Mach. Learn. Res. , vol. 17, no. 1, pp. 26572681, 2016. -
  33. [33][33] S. Niknam, H. S. Dhillon, and J. H. Reed, 'Federated learning for wireless communications: Motivation, opportunities and challenges,' arXiv preprint arXiv:1908.06847 , 2019. - · arXiv
  34. [34][34] G. Zhu, D. Liu, Y. Du, C. You, J. Zhang, and K. Huang, 'Toward an intelligent edge: Wireless communication meets machine learning,' IEEE Commun. Mag. , vol. 58, no. 1, pp. 19-25, 2020. -
  35. [35][35] C. L. P. Chen, G. Wen, Y. Liu, and F. Wang, 'Adaptive consensus control for a class of nonlinear multiagent time-delay systems using neural networks,' IEEE Trans. Neural Netw. Learning Syst. , vol. 25, no. 6, pp. 1217-1226, 2014. -
  36. [36][36] T. Li and J.-F. Zhang, 'Consensus conditions of multi-agent systems with time-varying topologies and stochastic communication noises,' IEEE Trans. Autom. Control , vol. 55, no. 9, pp. 2043-2057, 2010. -
  37. [37][37] S. Kar and J. M. F. Moura, 'Distributed consensus algorithms in sensor networks with imperfect communication: Link failures and channel noise,' IEEE Trans. Signal Process. , vol. 57, no. 1, pp. 355-369, 2009. -
  38. [38][38] S. Manfredi, 'Design of a multi-hop dynamic consensus algorithm over wireless sensor networks,' Control Eng. Practice , vol. 21, no. 4, pp. 381-394, 2013. -
  39. [39][39] A. Nedic and A. Ozdaglar, 'Distributed subgradient methods for multiagent optimization,' IEEE Trans. Auto. Control , vol. 54, no. 1, pp. 48-61, 2009. -
  40. [40][40] T. Chang, M. Hong, and X. Wang, 'Multi-agent distributed optimization via inexact consensus ADMM,' IEEE Trans. Signal Process. , vol. 63, no. 2, pp. 482-497, 2015. -
  41. [41][41] T. Chang, A. Nedi´ c, and A. Scaglione, 'Distributed constrained optimization by consensus-based primal-dual perturbation method,' IEEE Trans. Autom. Control , vol. 59, no. 6, pp. 1524-1538, 2014. -
  42. [42][42] B. Johansson, T. Keviczky, M. Johansson, and K. H. Johansson, 'Subgradient methods and consensus algorithms for solving convex optimization problems,' in Proc. IEEE Conf. Decis. Control , 2008, pp. 4185-4190. -
  43. [43][43] R. N. Clarke, 'Expanding mobile wireless capacity: The challenges presented by technology and economics,' Telecommun. Policy , vol. 38, no. 8-9, pp. 693-708, 2014. -
  44. [44][44] U. N. Kar and D. K. Sanyal, 'An overview of device-to-device communication in cellular networks,' ICT Express , vol. 4, no. 4, pp. 203-208, 2018. -
  45. [45][45] T. Zeng, O. Semiari, M. Mozaffari, M. Chen, W. Saad, and M. Bennis, 'Federated learning in the sky: Joint power allocation and scheduling with UAV swarms,' arXiv preprint arXiv:2002.08196 , 2020. - · arXiv
  46. [46][46] L. Xiao and S. Boyd, 'Fast linear iterations for distributed averaging,' Syst. & Control Lett. , vol. 53, no. 1, pp. 65-78, 2004. -
  47. [47][47] A. Reisizadeh, A. Mokhtari, H. Hassani, A. Jadbabaie, and R. Pedarsani, 'Fedpaq: A communication-efficient federated learning method with periodic averaging and quantization,' arXiv:1909.13014 , 2019. - · arXiv
  48. [48][48] C. Dinh, N. H. Tran, M. N. Nguyen, C. S. Hong, W. Bao, A. Zomaya, and V. Gramoli, 'Federated learning over wireless networks: Convergence analysis and resource allocation,' arXiv preprint arXiv:1910.13067 , 2019. - · arXiv
  49. [49][49] M. P. Friedlander and M. Schmidt, 'Hybrid deterministic-stochastic methods for data fitting,' SIAM J. Sci. Comput. , vol. 34, no. 3, pp. A1380-A1405, 2012. -
  50. [50][50] L. Xiao, S. Boyd, and S.-J. Kim, 'Distributed average consensus with least-mean-square deviation,' J. Parallel Distrib. Comput. , vol. 67, no. 1, pp. 33-46, 2007. -
  51. [51][51] X. Jia, 'Wireless networks and random geometric graphs,' in Proc. Int. Symp. Parallel Arch. Alg. Netw. , 2004, pp. 575-579. -
  52. [52][52] M. Hmila, M. Fern´ andez-Veiga, M. Rodr´ ıguez-P´ erez, and S. Herrer´ ıaAlonso, 'Energy efficient power and channel allocation in underlay device to multi device communications,' IEEE Trans. Commun. , vol. 67, no. 8, pp. 5817-5832, 2019. -
  53. [54][54] C. Xie, S. Koyejo, and I. Gupta, 'Asynchronous federated optimization,' arXiv preprint arXiv:1903.03934 , 2019. - · arXiv
  54. [55][55] B. T. Polyak, 'Gradient methods for minimizing functionals,' Zh. Vychisl. Mat. Mat. Fiz , vol. 3, no. 4, pp. 643-653, 1963. -

Cited by

Sorted by citations

Authors
Hosseinalipour, S., Azam, S. S., Brinton, C. G., Michelusi, N., Aggarwal, V., Love, D. J., & Dai, H.
Affiliations
  • Purdue University West Lafayette (US)
  • North Carolina State University (US)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2020
Volume
30
Issue
4
Pages
1569-1584
ISSN
1063-6692 (ISSN-L)
DOI
10.48550/arxiv.2007.09511
arXiv
2007.09511
OpenAlex
W3094368349
Cited by
131
References
0
Citations by year
2026: 15 · 2025: 23 · 2024: 38 · 2023: 22 · 2022: 19 · 2021: 11 · 2020: 3
FWCI
8.36
Type
journal article
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • IEEE/ACM Transactions on Networking (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Funders
  • National Science Foundation (1941529, 1642982, 2129015, 1824518)
  • Office of Naval Research (N000142112472)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Privacy-Preserving Technologies in Data
  • Physical Sciences › Computer Science › Computer Networks and Communications › IoT and Edge/Fog Computing
  • Physical Sciences › Computer Science › Artificial Intelligence › Stochastic Gradient Optimization Techniques
Keywords
device-to-device communication, fog computing, convergence analysis, spectral radius, distributed control algorithms

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review