Decentralised Learning in Federated Deployment Environments: A System-Level Survey
Open accessThe DOI matches its Crossref record; 51 of the 51 references it cites were found in a registry or an index.
Abstract
Decentralised learning is attracting more and more interest because it embodies the principles of data minimisation and focused data collection, while favouring the transparency of purpose specification (i.e., the objective for which a model is built). Cloud-centric-only processing and deep learning are no longer strict necessities to train high-fidelity models; edge devices can actively participate in the decentralised learning process by exchanging meta-level information in place of raw data, thus paving the way for better privacy guarantees. In addition, these new possibilities can relieve the network backbone from unnecessary data transfer and allow it to meet strict low-latency requirements by leveraging on-device model inference. This survey provides a detailed and up-to-date overview of the most recent contributions available in the state-of-the-art decentralised learning literature. In particular, it originally provides the reader audience with a clear presentation of the peculiarities of federated settings, with a novel taxonomy of decentralised learning approaches, and with a detailed description of the most relevant and specific system-level contributions of the surveyed solutions for privacy, communication efficiency, non-IIDness, device heterogeneity, and poisoning defense.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
New directions in cryptography
Diffie, W., & Hellman, M. (1976) · IEEE Transactions on Information Theory · doi:10.1109/tit.1976.1055638 · Cited by 14,713
Shamir, A. (1979) · Communications of the ACM · doi:10.1145/359168.359176 · Cited by 13,756 · cited 2×
“The key idea is to partition the federation of learners in groups that actively participate in the aggregation and dropout-recovery phases instead of just leveraging the central server, and to add redundancy directly in the model updates to reconstruct the missing contributions of dropout participants instead of Shami…”
MethodFederated Machine Learning: Concept and Applications
Yang, Q., Liu, Y., Chen, T., & Tong, Y. (2019) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3298981 · Cited by 6,322
Federated learning: Challenges, methods, and future directions
Li, T., Sahu, A. K., Talwalkar, A., & Smith, V. (2020) · IEEE Signal Processing Magazine · doi:10.1109/msp.2020.2975749 · Cited by 6,210 · cited 2×
“(3) Our survey includes several very recent research papers (published in the last few months) that are relevant for the community and not covered yet by [68] and [129].”
BackgroundFinite-time Analysis of the Multiarmed Bandit Problem
Auer, P., Cesa-Bianchi, N., & Fischer, P. (2002) · Machine Learning · doi:10.1023/a:1013689704352 · Cited by 6,033
The Algorithmic Foundations of Differential Privacy
Dwork, C., & Roth, A. (2014) · Foundations and Trends® in Theoretical Computer Science · doi:10.1561/0400000042 · Cited by 5,557
Federated Optimization in Heterogeneous Networks
Li, T., Sahu, A. K., Zaheer, M., Sanjabi, M., Talwalkar, A., & Smith, V. (2018) · arXiv · doi:10.48550/arxiv.1812.06127 · Cited by 4,023
Practical Secure Aggregation for Privacy-Preserving Machine Learning
Bonawitz, K., Ivanov, V., Kreuter, B., Marcedone, A., McMahan, H. B., Patel, S., Ramage, D., Segal, A., & Seth, K. (2017) · Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security · doi:10.1145/3133956.3133982 · Cited by 3,956 · cited 6×
“Furthermore, other aspects in conjugating privacy and communication efficiency emerge; the secure aggregation protocol [13] can be redesigned to account from the beginning for communication efficiency [14], while tailored DP-mechanisms can be more amenable to privacy analysis when quantisation of…”
BackgroundMethodInfluentialDeep Learning Based Recommender System: A Survey and New Perspectives
Zhang, S., Yao, L., Sun, A., & Tay, Y. (2020) · ACM Computing Surveys · doi:10.1145/3285029 · Cited by 3,485
Deep learning and its applications to machine health monitoring
Zhao, R., Yan, R., Chen, Z., Mao, K., Wang, P., & Gao, R. X. (2019) · Mechanical Systems and Signal Processing · doi:10.1016/j.ymssp.2018.05.050 · Cited by 2,894
Measuring and testing dependence by correlation of distances
Székely, G. J., Rizzo, M. L., & Bakirov, N. K. (2007) · The Annals of Statistics · doi:10.1214/009053607000000505 · Cited by 2,684 · cited 1×
“[123] hence train the neural network by using a weighted combination of two losses as loss function, and such losses are the log distance correlation [116] and the categorical cross-entropy.”
MethodEdge Intelligence: Paving the Last Mile of Artificial Intelligence With Edge Computing
Zhou, Z., Chen, X., Li, E., Zeng, L., Luo, K., & Zhang, J. (2019) · Proceedings of the IEEE · doi:10.1109/jproc.2019.2918951 · Cited by 2,455 · cited 3×
“(5) Finally, differently from [73] and [147], we do not specifically focus only on the advances of decentralised learning that can be achieved via Multi-access Edge Computing (MEC).”
BackgroundAdaptive Federated Learning in Resource Constrained Edge Computing Systems
Wang, S., Tuor, T., Salonidis, T., Leung, K. K., Makaya, C., He, T., & Chan, K. (2019) · IEEE Journal on Selected Areas in Communications · doi:10.1109/jsac.2019.2904348 · Cited by 2,232
Federated Learning with Non-IID Data
Zhao, Y., Li, M., Lai, L., Suda, N., Civin, D., & Chandra, V. (2018) · arXiv · doi:10.48550/arxiv.1806.00582 · Cited by 2,077
Zhu, L., & Han, S. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-63076-8_2 · Cited by 1,906
Client Selection for Federated Learning with Heterogeneous Resources in Mobile Edge
Nishio, T., & Yonetani, R. (2019) · ICC 2019 - 2019 IEEE International Conference on Communications (ICC) · doi:10.1109/icc.2019.8761315 · Cited by 1,875 · cited 3×
“[91] propose to extend the FL algorithm by replacing the random selection of clients with a two-step client selection; the MEC operator asks random clients to provide their resource information (computational capacities, wireless channel states, size of the dataset relevant to the current training task) from which dec…”
BackgroundClient Selection for Federated Learning with Heterogeneous Resources in Mobile Edge
Nishio, T., & Yonetani, R. (2018) · arXiv · doi:10.48550/arxiv.1804.08333 · Cited by 1,875
SCAFFOLD: Stochastic Controlled Averaging for Federated Learning
Karimireddy, S. P., Kale, S., Mohri, M., Reddi, S. J., Stich, S. U., & Suresh, A. T. (2019) · arXiv · doi:10.48550/arxiv.1910.06378 · Cited by 1,734 · cited 8×
“SCAFFOLD [56] 2019 device Full w , c w , c ×”
BackgroundInfluentialExploiting Unintended Feature Leakage in Collaborative Learning
Melis, L., Song, C., De Cristofaro, E., & Shmatikov, V. (2019) · 2019 IEEE Symposium on Security and Privacy (SP) · doi:10.1109/sp.2019.00029 · Cited by 1,734
Robust and Communication-Efficient Federated Learning From Non-i.i.d. Data
Sattler, F., Wiedemann, S., Muller, K.-R., & Samek, W. (2020) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2019.2944481 · Cited by 1,725
Privacy-Preserving Deep Learning via Additively Homomorphic Encryption
Phong, L. T., Aono, Y., Hayashi, T., Wang, L., & Moriai, S. (2018) · IEEE Transactions on Information Forensics and Security · doi:10.1109/tifs.2017.2787987 · Cited by 1,662
Federated Optimization: Distributed Machine Learning for On-Device Intelligence
Konečný, J., McMahan, H. B., Ramage, D., & Richtárik, P. (2016) · arXiv · doi:10.48550/arxiv.1610.02527 · Cited by 1,563
Deep Models Under the GAN: Information Leakage from Collaborative Deep Learning
Hitaj, B., Ateniese, G., & Perez-Cruz, F. (2017) · Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security · doi:10.1145/3133956.3134012 · Cited by 1,377 · cited 1×
“However, it has been demonstrated that gradients exchanged during the distributed training process do leak information about the training data [40, 45, 89, 97, 140, 148] as well as model updates [84, 89]—even though it may be preferable to exchange model weights instead of gradients under a privacypreserving perspecti…”
BackgroundFederated Learning for Mobile Keyboard Prediction
Hard, A., Rao, K., Mathews, R., Ramaswamy, S., Beaufays, F., Augenstein, S., Eichner, H., Kiddon, C., & Ramage, D. (2018) · arXiv · doi:10.48550/arxiv.1811.03604 · Cited by 1,148 · cited 1×
“Related examples include on-device intelligent keyboards that power content suggestions [130], or that predict the most suitable next words [38] or the most fitting Emojis [100] given the chat history; or again vocabularies that evolve to follow the ongoing trending expressions by learning out-of-vocabulary words [18]…”
BackgroundDeep Gradient Compression: Reducing the Communication Bandwidth for Distributed Training
Lin, Y., Han, S., Mao, H., Wang, Y., & Dally, W. J. (2017) · arXiv · doi:10.48550/arxiv.1712.01887 · Cited by 1,134 · cited 1×
“degrading the resulting model accuracy [3, 74, 115].”
A Hybrid Approach to Privacy-Preserving Federated Learning
Truex, S., Baracaldo, N., Anwar, A., Steinke, T., Ludwig, H., Zhang, R., & Zhou, Y. (2019) · Proceedings of the 12th ACM Workshop on Artificial Intelligence and Security · doi:10.1145/3338501.3357370 · Cited by 1,029
Federated learning of predictive models from federated Electronic Health Records
Brisimi, T. S., Chen, R., Mela, T., Olshevsky, A., Paschalidis, I. C., & Shi, W. (2018) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2018.01.007 · Cited by 997 · cited 1×
“Other clinical learning tasks include prediction of prolonged length of stay and in-hospital mortality [96], prediction of hospitalisations for cardiac events [15], or gaining insights about brain diseases [104].”
BackgroundBlockchained On-Device Federated Learning
Kim, H., Park, J., Bennis, M., & Kim, S.-L. (2018) · arXiv · doi:10.48550/arxiv.1808.03949 · Cited by 882 · cited 1×
“, [58]), though not being explored in this survey.”
BackgroundFederated Learning for Wireless Communications: Motivation, Opportunities, and Challenges
Niknam, S., Dhillon, H. S., & Reed, J. H. (2020) · IEEE Communications Magazine · doi:10.1109/mcom.001.1900461 · Cited by 820 · cited 1×
“Decentralised learning has been used also to conjugate user privacy and prediction ability of the infrastructure in the 5G multi-access edge computing architecture [24, 57, 80], for example for proactive content caching [135] or for optimal allocation of virtual machine replicas copies [31], and it is considered a key…”
Distributed learning of deep neural network over multiple agents
Gupta, O., & Raskar, R. (2018) · Journal of Network and Computer Applications · doi:10.1016/j.jnca.2018.05.003 · Cited by 726
Applied Federated Learning: Improving Google Keyboard Query Suggestions
Yang, T., Andrew, G., Eichner, H., Sun, H., Li, W., Kong, N., Ramage, D., & Beaufays, F. (2018) · arXiv · doi:10.48550/arxiv.1812.02903 · Cited by 684 · cited 1×
“Related examples include on-device intelligent keyboards that power content suggestions [130], or that predict the most suitable next words [38] or the most fitting Emojis [100] given the chat history; or again vocabularies that evolve to follow the ongoing trending expressions by learning out-of-vocabulary words [18]…”
BackgroundYang (2019) · doi:10.1145/3339474 · Cited by 632
Sparse Communication for Distributed Gradient Descent
Aji, A. F., & Heafield, K. (2017) · Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d17-1045 · Cited by 585
signSGD: Compressed Optimisation for Non-Convex Problems
Bernstein, J., Wang, Y.-X., Azizzadenesheli, K., & Anandkumar, A. (2018) · arXiv · doi:10.48550/arxiv.1802.04434 · Cited by 541
Split learning for health: Distributed deep learning without sharing raw patient data
Vepakomma, P., Gupta, O., Swedish, T., & Raskar, R. (2018) · arXiv · doi:10.48550/arxiv.1812.00564 · Cited by 513
Communication-Efficient Federated Learning for Wireless Edge Intelligence in IoT
Mills, J., Hu, J., & Min, G. (2020) · IEEE Internet of Things Journal · doi:10.1109/jiot.2019.2956615 · Cited by 436 · cited 7×
“[85] highlight that this is compensated by the faster convergence of CE-FedAvg.”
BackgroundMethodInfluentialFederated Learning With Cooperating Devices: A Consensus Approach for Massive IoT Networks
Savazzi, S., Nicoli, M., & Rampa, V. (2020) · IEEE Internet of Things Journal · doi:10.1109/jiot.2020.2964162 · Cited by 431
Accelerating Federated Learning via Momentum Gradient Descent
Liu, W., Chen, L., Chen, Y., & Zhang, W. (2020) · IEEE Transactions on Parallel and Distributed Systems · doi:10.1109/tpds.2020.2975189 · Cited by 374 · cited 3×
“Momentum FL [77] 2020 device Full w , d w , d ×”
BackgroundTurbo-Aggregate: Breaking the Quadratic Aggregation Barrier in Secure Federated Learning
So, J., Guler, B., & Avestimehr, A. S. (2021) · IEEE Journal on Selected Areas in Information Theory · doi:10.1109/jsait.2021.3054610 · Cited by 318 · cited 3×
“A recent work [112], namely Turbo-Aggregate, addresses the quadratic growth of the computational cost and of the communication overhead by slightly changing the approach, and still being resilient to user dropouts (up to 50% of participants).”
BackgroundMethodFederated Learning for Keyword Spotting
Leroy, D., Coucke, A., Lavril, T., Gisselbrecht, T., & Dureau, J. (2019) · ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2019.8683546 · Cited by 291
Sparse Binary Compression: Towards Distributed Deep Learning with minimal Communication
Sattler, F., Wiedemann, S., Muller, K.-R., & Samek, W. (2019) · 2019 International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.2019.8852172 · Cited by 240
Federated Learning for Emoji Prediction in a Mobile Keyboard
Ramaswamy, S., Mathews, R., Rao, K., & Beaufays, F. (2019) · arXiv · doi:10.48550/arxiv.1906.04329 · Cited by 231 · cited 1×
“Related examples include on-device intelligent keyboards that power content suggestions [130], or that predict the most suitable next words [38] or the most fitting Emojis [100] given the chat history; or again vocabularies that evolve to follow the ongoing trending expressions by learning out-of-vocabulary words [18]…”
BackgroundAdaptive bound optimization for online convex optimization
McMahan, H. B., & Streeter, M. (2010) · arXiv · doi:10.48550/arxiv.1002.4908 · Cited by 227
Threats to federated learning A survey
Lyu, L., Yu, H., & Yang, Q. (2020) · arXiv · doi:10.48550/arxiv.2003.02133 · Cited by 221 · cited 1×
“We redirect the reader to [79] for a complete understanding of the threat model and of the attack variety.”
BackgroundDifferential Privacy Has Disparate Impact on Model Accuracy
Bagdasaryan, E., & Shmatikov, V. (2019) · arXiv · doi:10.48550/arxiv.1905.12101 · Cited by 219 · cited 1×
“The literature about protecting privacy in decentralised learning comprises diverse approaches; differentially private mechanisms [34, 82] can be employed during the distributed training process to mask updates at the cost of reduced model accuracy [7], and relaxations of traditional Differential Privacy (DP) can be l…”
BackgroundHuman Activity Recognition Using Federated Learning
Sozinov, K., Vlassov, V., & Girdzijauskas, S. (2018) · 2018 IEEE Intl Conf on Parallel & Distributed Processing with Applications, Ubiquitous Computing & Communications, Big Data & Cloud Computing, Social Computing & Networking, Sustainable Computing & Communications (ISPA/IUCC/BDCloud/SocialCom/SustainCom) · doi:10.1109/bdcloud.2018.00164 · Cited by 167 · cited 1×
“, [113]) and keyword spotting for voice assistants in smart homes (e.”
BackgroundFederated Learning Based Proactive Content Caching in Edge Computing
Yu, Z., Hu, J., Min, G., Lu, H., Zhao, Z., Wang, H., & Georgalas, N. (2018) · 2018 IEEE Global Communications Conference (GLOBECOM) · doi:10.1109/glocom.2018.8647616 · Cited by 140 · cited 1×
“Decentralised learning has been used also to conjugate user privacy and prediction ability of the infrastructure in the 5G multi-access edge computing architecture [24, 57, 80], for example for proactive content caching [135] or for optimal allocation of virtual machine replicas copies [31], and it is considered a key…”
Abnormal client behavior detection in federated learning
Li, S., Cheng, Y., Liu, Y., Wang, W., & Chen, T. (2019) · arXiv · doi:10.48550/arxiv.1910.09933 · Cited by 122 · cited 1×
“[66] use a server-side pre-trained autoencoder model to detect abnormal weight updates that are then accordingly penalised during the aggregation.”
MethodFederated Learning Of Out-Of-Vocabulary Words
Chen, M., Mathews, R., Ouyang, T., & Beaufays, F. (2019) · arXiv · doi:10.48550/arxiv.1903.10635 · Cited by 97 · cited 1×
“Related examples include on-device intelligent keyboards that power content suggestions [130], or that predict the most suitable next words [38] or the most fitting Emojis [100] given the chat history; or again vocabularies that evolve to follow the ongoing trending expressions by learning out-of-vocabulary words [18]…”
BackgroundBACombo—Bandwidth-Aware Decentralized Federated Learning
Jiang, J., Hu, L., Hu, C., Liu, J., & Wang, Z. (2020) · Electronics · doi:10.3390/electronics9030440 · Cited by 47
Federated and Differentially Private Learning for Electronic Health Records
Pfohl, S. R., Dai, A. M., & Heller, K. (2019) · arXiv · doi:10.48550/arxiv.1911.05861 · Cited by 45 · cited 1×
“Other clinical learning tasks include prediction of prolonged length of stay and in-hospital mortality [96], prediction of hospitalisations for cardiac events [15], or gaining insights about brain diseases [104].”
Background
Cited by
Federated learning for healthcare: Systematic review and architecture proposal
Antunes, R. S., André da Costa, C., Küderle, A., Yari, I. A., & Eskofier, B. (2022) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3501813 · Cited by 595
Blockchain-empowered Federated Learning: Challenges, Solutions, and Future Directions
Zhu, J., Cao, J., Saxena, D., Jiang, S., & Ferradi, H. (2023) · ACM Computing Surveys · doi:10.1145/3570953 · Cited by 309
AI Models for Green Communications Towards 6G
Mao, B., Tang, F., Kawamoto, Y., & Kato, N. (2022) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2021.3130901 · Cited by 300
A Comprehensive Survey on Poisoning Attacks and Countermeasures in Machine Learning
Tian, Z., Cui, L., Liang, J., & Yu, S. (2023) · ACM Computing Surveys · doi:10.1145/3551636 · Cited by 271
TrustFed: A Framework for Fair and Trustworthy Cross-Device Federated Learning in IIoT
Rehman, M. H. U., Dirir, A. M., Salah, K., Damiani, E., & Svetinovic, D. (2021) · IEEE Transactions on Industrial Informatics · doi:10.1109/tii.2021.3075706 · Cited by 132
Metadata
- Authors
- Bellavista, P., Foschini, L., & Mora, A.
- Venue
- ACM Computing Surveys
- Published
- 2021-02-11
- Volume
- 54
- Issue
- 1
- Pages
- 1-38
- ISSN
- 0360-0300 (ISSN-L)
- 1557-7341
- OpenAlex
- W3129828944
- Cited by
- 99
- References
- 51
- FWCI
- 5.91
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.