Multi-Task Federated Learning for Personalised Deep Neural Networks in Edge Computing
Open accessThe DOI matches its Crossref record; 33 of the 33 references it cites were found in a registry or an index.
Abstract
Federated Learning (FL) is an emerging approach for collaboratively training Deep Neural Networks (DNNs) on mobile devices, without private user data leaving the devices. Previous works have shown that non-Independent and Identically Distributed (non-IID) user data harms the convergence speed of the FL algorithms. Furthermore, most existing work on FL measures global-model accuracy, but in many cases, such as user content-recommendation, improving individual User model Accuracy (UA) is the real objective. To address these issues, we propose a Multi-Task FL (MTFL) algorithm that introduces non-federated Batch-Normalization (BN) layers into the federated DNN. MTFL benefits UA and convergence speed by allowing users to train models personalised to their own data. MTFL is compatible with popular iterative FL optimisation algorithms such as Federated Averaging (FedAvg), and we show empirically that a distributed form of Adam optimisation (FedAvg-Adam) benefits convergence speed even further when used as the optimisation strategy within MTFL. Experiments using MNIST and CIFAR10 demonstrate that MTFL is able to significantly reduce the number of rounds required to reach a target UA, by up to$5\times$when using existing FL optimisation strategies, and with a further$3\times$improvement when using FedAvg-Adam. We compare MTFL to competing personalised FL algorithms, showing that it is able to achieve the best UA for MNIST and CIFAR10 in all considered scenarios. Finally, we evaluate MTFL with FedAvg-Adam on an edge-computing testbed, showing that its convergence and UA benefits outweigh its overhead.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
Learning Multiple Layers of Features from Tiny Images
Krizhevsky, A. (2024) · doi:10.57702/zp44cu3g · Cited by 23,395
Communication-Efficient Learning of Deep Networks from Decentralized Data
McMahan, H. B., Moore, E., Ramage, D., Hampson, S., & Arcas, B. A. Y. (2016) · arXiv · doi:10.48550/arxiv.1602.05629 · Cited by 14,215
Federated learning: Challenges, methods, and future directions
Li, T., Sahu, A. K., Talwalkar, A., & Smith, V. (2020) · IEEE Signal Processing Magazine · doi:10.1109/msp.2020.2975749 · Cited by 6,210
A Survey on Mobile Edge Computing: The Communication Perspective
Mao, Y., You, C., Zhang, J., Huang, K., & Letaief, K. B. (2017) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2017.2745201 · Cited by 5,607 · cited 8×
“Finally, we evaluate the cost in terms of computation time of MTFL(FedAvg-Adam) on an MEC-like testbed.”
BackgroundInfluentialFederated Optimization in Heterogeneous Networks
Li, T., Sahu, A. K., Zaheer, M., Sanjabi, M., Talwalkar, A., & Smith, V. (2018) · arXiv · doi:10.48550/arxiv.1812.06127 · Cited by 4,023
Federated Learning with Non-IID Data
Zhao, Y., Li, M., Lai, L., Suda, N., Civin, D., & Chandra, V. (2018) · arXiv · doi:10.48550/arxiv.1806.00582 · Cited by 2,077
Client Selection for Federated Learning with Heterogeneous Resources in Mobile Edge
Nishio, T., & Yonetani, R. (2019) · ICC 2019 - 2019 IEEE International Conference on Communications (ICC) · doi:10.1109/icc.2019.8761315 · Cited by 1,875
Client Selection for Federated Learning with Heterogeneous Resources in Mobile Edge
Nishio, T., & Yonetani, R. (2018) · arXiv · doi:10.48550/arxiv.1804.08333 · Cited by 1,875
Federated Learning for Mobile Keyboard Prediction
Hard, A., Rao, K., Mathews, R., Ramaswamy, S., Beaufays, F., Augenstein, S., Eichner, H., Kiddon, C., & Ramage, D. (2018) · arXiv · doi:10.48550/arxiv.1811.03604 · Cited by 1,148
Adaptive Federated Optimization
Reddi, S., Charles, Z., Zaheer, M., Garrett, Z., Rush, K., Konečný, J., Kumar, S., & McMahan, H. B. (2020) · arXiv · doi:10.48550/arxiv.2003.00295 · Cited by 840
Personalized Cross-Silo Federated Learning on Non-IID Data
Huang, Y., Chu, L., Zhou, Z., Wang, L., Liu, J., Pei, J., & Zhang, Y. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i9.16960 · Cited by 723
Scheduling Policies for Federated Learning in Wireless Networks
Yang, H. H., Liu, Z., Quek, T. Q. S., & Poor, H. V. (2020) · IEEE Transactions on Communications · doi:10.1109/tcomm.2019.2944169 · Cited by 640
Personalized Federated Learning with Theoretical Guarantees: A Model-Agnostic Meta-Learning Approach
Fallah, A., Mokhtari, A., & Ozdaglar, A. (2020) · neural information processing systems · Cited by 596
CMFL: Mitigating Communication Overhead for Federated Learning
WANG, L., WANG, W., & LI, B. (2019) · 2019 IEEE 39th International Conference on Distributed Computing Systems (ICDCS) · doi:10.1109/icdcs.2019.00099 · Cited by 475
Huang, L., Shea, A. L., Qian, H., Masurkar, A., Deng, H., & Liu, D. (2019) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2019.103291 · Cited by 437
Self-Balancing Federated Learning With Global Imbalanced Data in Mobile Systems
Duan, M., Liu, D., Chen, X., Liu, R., Tan, Y., & Liang, L. (2021) · IEEE Transactions on Parallel and Distributed Systems · doi:10.1109/tpds.2020.3009406 · Cited by 432
Improving Federated Learning Personalization via Model Agnostic Meta Learning
Jiang, Y., Konečný, J., Rush, K., & Kannan, S. (2019) · arXiv · doi:10.48550/arxiv.1909.12488 · Cited by 381 · cited 2×
“[11] highlighted the connection between FedAvg and first-order MAML updates, and proposed a”
BackgroundAccelerating Federated Learning via Momentum Gradient Descent
Liu, W., Chen, L., Chen, Y., & Zhang, W. (2020) · IEEE Transactions on Parallel and Distributed Systems · doi:10.1109/tpds.2020.2975189 · Cited by 374
Salvaging Federated Learning by Local AdaptationPreprint
Yu, T., Bagdasaryan, E., & Shmatikov, V. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2002.04758 · Cited by 338
Federated Learning for Keyword Spotting
Leroy, D., Coucke, A., Lavril, T., Gisselbrecht, T., & Dureau, J. (2019) · ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2019.8683546 · Cited by 291
Federated Learning of a Mixture of Global and Local Models
Hanzely, F., & Richtárik, P. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2002.05516 · Cited by 230
BrainTorrent: A Peer-to-Peer Environment for Decentralized Federated Learning
Roy, A. G., Siddiqui, S., Pölsterl, S., Navab, N., & Wachinger, C. (2019) · arXiv · doi:10.48550/arxiv.1905.06731 · Cited by 212
Computation Offloading in Multi-Access Edge Computing: A Multi-Task Learning Approach
Yang, B., Cao, X., Bassey, J., Li, X., & Qian, L. (2021) · IEEE Transactions on Mobile Computing · doi:10.1109/tmc.2020.2990630 · Cited by 204
Federated Collaborative Filtering for Privacy-Preserving Personalized Recommendation System
Ammad-ud-din, M., Ivannikova, E., Khan, S. A., Oyomno, W., Fu, Q., Tan, K. E., & Flanagan, A. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1901.09888 · Cited by 203
Learning-Aided Computation Offloading for Trusted Collaborative Mobile Edge Computing
Li, Y., Wang, X., Gan, X., Jin, H., Fu, L., & Wang, X. (2020) · IEEE Transactions on Mobile Computing · doi:10.1109/tmc.2019.2934103 · Cited by 167
Wireless Federated Distillation for Distributed Edge Learning with Heterogeneous Data
Ahn, J.-H., Simeone, O., & Kang, J. (2019) · IEEE International Symposium on Personal, Indoor and Mobile Radio Communications (PIMRC) · doi:10.1109/pimrc.2019.8904164 · Cited by 150
Wang, C., Yang, Y., & Zhou, P. (2020) · IEEE Transactions on Parallel and Distributed Systems · doi:10.1109/tpds.2020.3023905 · Cited by 118
K for the Price of 1: Parameter-efficient Multi-task and Transfer LearningPreprint
Mudrakarta, P. K., Sandler, M., Zhmoginov, A., & Howard, A. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1810.10703 · Cited by 76
Robust Learning from Untrusted SourcesPreprint
Konstantinov, N., & Lampert, C. H. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1901.10310 · Cited by 57
Computation Offloading in Multi-Access Edge Computing Networks: A Multi-Task Learning Approach
Yang, B. F., Cao, X., Bassey, J., Li, X., Kroecker, T. S., & Qian, L. (2019) · International Conference on Communications · doi:10.1109/icc.2019.8761212 · Cited by 42
FedU: A Unified Framework for Federated Multi-Task Learning with Laplacian Regularization.Preprint
Dinh, C. T., Vu, T. T., Tran, N. H. K., Dao, M. N., & Zhang, H. (2021) · arXiv (Cornell University) · Cited by 39
Distributed Deep Learning Optimized System over the Cloud and Smart Phone Devices
Jiang, H., Starkman, J., Lee, Y., Chen, H., Qian, X., & Huang, M.-C. (2019) · IEEE Transactions on Mobile Computing · doi:10.1109/tmc.2019.2941492 · Cited by 31
Cited by
A Survey on Metaverse: Fundamentals, Security, and Privacy
Wang, Y., Su, Z., Zhang, N., Xing, R., Liu, D., Luan, T. H., & Shen, X. (2023) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2022.3202047 · Cited by 1,575
A survey on federated learning: challenges and applications
Wen, J., Zhang, Z., Lan, Y., Cui, Z., Cai, J., & Zhang, W. (2023) · International Journal of Machine Learning and Cybernetics · doi:10.1007/s13042-022-01647-y · Cited by 883
Federated Learning With Non-IID Data: A Survey
Lu, Z., Pan, H., Dai, Y., Si, X., & Zhang, Y. (2024) · IEEE Internet of Things Journal · doi:10.1109/jiot.2024.3376548 · Cited by 430
Security and Privacy on 6G Network Edge: A Survey
Mao, B., Liu, J., Wu, Y., & Kato, N. (2023) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2023.3244674 · Cited by 321
On Model Transmission Strategies in Federated Learning With Lossy Communications
Su, X., Zhou, Y., Cui, L., & Liu, J. (2023) · IEEE Transactions on Parallel and Distributed Systems · doi:10.1109/tpds.2023.3240883 · Cited by 47
Metadata
- Authors
- Mills, J., Hu, J., & Min, G.
- Venue
- IEEE Transactions on Parallel and Distributed Systems
- Published
- 2022-03-01
- Volume
- 33
- Issue
- 3
- Pages
- 630-641
- ISSN
- 1045-9219 (ISSN-L)
- 1558-2183
- 2161-9883
- OpenAlex
- W3186051974
- Cited by
- 303
- References
- 33
- FWCI
- 17.64
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.