Averaging Is Probably Not the Optimum Way of Aggregating Parameters in Federated Learning
Open accessAbstract
Federated learning is a decentralized topology of deep learning, that trains a shared model through data distributed among each client (like mobile phones, wearable devices), in order to ensure data privacy by avoiding raw data exposed in data center (server). After each client computes a new model parameter by stochastic gradient descent (SGD) based on their own local data, these locally-computed parameters will be aggregated to generate an updated global model. Many current state-of-the-art studies aggregate different client-computed parameters by averaging them, but none theoretically explains why averaging parameters is a good approach. In this paper, we treat each client computed parameter as a random vector because of the stochastic properties of SGD, and estimate mutual information between two client computed parameters at different training phases using two methods in two learning tasks. The results confirm the correlation between different clients and show an increasing trend of mutual information with training iteration. However, when we further compute the distance between client computed parameters, we find that parameters are getting more correlated while not getting closer. This phenomenon suggests that averaging parameters may not be the optimum way of aggregating trained parameters.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Decentralized training is more efficient than centralized training when running federated systems in low-bandwidth or high-latency networks.
“Decentralized training has also been shown to be more efficient than centralized training when running federated systems in low-bandwidth or high-latency networks (Xiao et al. 2020;Liu et al. 2020;Jiang et al. 2020).”
From Survey: federated learning data security and privacy-preserving in edge-Internet of Things (2024) · cites this paper as Xiao et al. 2020
The Introduction describes federated learning as training on distributed client data, motivated by privacy concerns rather than a measured efficiency advantage. The Conclusions suggest that compression could improve communication efficiency, but neither section compares decentralized with centralized training in low-bandwidth or high-latency networks. Searches for those network conditions and the claimed efficiency comparison found no supporting passage. (LLM, conf=0.73)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155 · cited 1× · see citing works
“We firstly calculate 61 the MI between different parameters in two learning tasks: (1) convolutional neural network (CNN) 62 [27] designed for a classification task and (2) recurrent neural network (RNN) [28] on a regression 63 task.”
MethodA Mathematical Theory of Communication
Shannon, C. E. (1948) · Bell System Technical Journal · doi:10.1002/j.1538-7305.1948.tb01338.x · Cited by 88,486 · see citing works
Gradient-based learning applied to document recognition
Lecun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998) · Proceedings of the IEEE · doi:10.1109/5.726791 · Cited by 58,805 · cited 1× · see citing works
“In the second experiment, we firstly train the whole Lenet model[35] on the MNIST dataset, 124 and then apply transfer learning by keeping the trained Lenet model unchanged except the last 125 fully connected layer and apply it on augmented data.”
MethodElements of Information Theory
Cover, T. M., & Thomas, J. A. (2001) · Wiley eBooks · doi:10.1002/0471200611 · Cited by 37,139 · see citing works
Dropout as a bayesian approximation: Representing model uncertainty in deep learning
Gal, Y., & Ghahramani, Z. (2015) · arXiv · doi:10.48550/arxiv.1506.02142 · Cited by 6,471 · cited 1× · see citing works
“The model architecture we use is a simple RNN with 10 layers and four units, where each layer is wrapped by Dropout function [38], for a total of 394 parameters.”
MethodFog computing and its role in the internet of things
Bonomi, F., Milito, R., Zhu, J., & Addepalli, S. (2012) · Proceedings of the first edition of the MCC workshop on Mobile cloud computing · doi:10.1145/2342509.2342513 · Cited by 6,244 · see citing works
Large-Scale Machine Learning with Stochastic Gradient Descent
Bottou, L. (2010) · Proceedings of COMPSTAT'2010 · doi:10.1007/978-3-7908-2604-3_16 · Cited by 5,795 · cited 3× · see citing works
“In this paper, we treat each client’s computed parameter vector wq as an random vector because of the stochastic properties of SGD, and train each client in one training epoch many times to obtain a series of parameter samples.”
BackgroundInfluentialRecurrent neural network based language model
Mikolov, T., Karafiát, M., Burget, L., Černocký, J., & Khudanpur, S. (2010) · Interspeech 2010 · doi:10.21437/interspeech.2010-343 · Cited by 5,390 · cited 4× · see citing works
“In the first experiment, we consider training an RNN model to predict the shift and scale variation of an arbitrary randomized 1-d signal sequence.”
MethodInfluentialThe MNIST Database of Handwritten Digit Images for Machine Learning Research [Best of the Web]
Deng, L. (2012) · IEEE Signal Processing Magazine · doi:10.1109/msp.2012.2211477 · Cited by 4,858 · see citing works
Kraskov, A., Stögbauer, H., & Grassberger, P. (2004) · Physical Review E · doi:10.1103/physreve.69.066138 · Cited by 4,157 · cited 4× · see citing works
“By using two methods, one is base on the MI derivative formula under the multi-dimensional Gaussian distribution assumption, and another is a continuous-continuous MI estimator based on k-nearest neighbor (KNN) distances described in [21].”
BackgroundMethodInfluentialLarge Scale Distributed Deep Networks
Dean, J. B., Corrado, G. S., Monga, R., Chen, K., Devin, M., Mao, M., Ranzato, M., Senior, A., Tucker, P. A., Yang, K., et al. (2012) · Cited by 3,189 · see citing works
Deep learning and the information bottleneck principle
Tishby, N., & Zaslavsky, N. (2015) · Information Theory Workshop · doi:10.1109/itw.2015.7133169 · Cited by 2,170 · cited 2× · see citing works
“In [29], they extend the approach of [28] demonstrating the effectiveness of the Information Plane visualization of DNNs.”
BackgroundDanielsson, P. (1980) · Computer Graphics and Image Processing · doi:10.1016/0146-664x(80)90054-4 · Cited by 2,132 · cited 1× · see citing works
“We further calculate the distance variation between different parameters using three regular distance metrics: Euclidean distance [25], Manhattan distance [26], Chebyshev distance [27].”
MethodRobust and Communication-Efficient Federated Learning From Non-i.i.d. Data
Sattler, F., Wiedemann, S., Muller, K.-R., & Samek, W. (2020) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2019.2944481 · Cited by 1,725 · cited 1× · see citing works
“In [14], they compress the model in both upstream and downstream links to promote communication efficiency.”
BackgroundOpening the Black Box of Deep Neural Networks via InformationPreprint
Shwartz-Ziv, R., & Tishby, N. (2017) · arXiv.org · doi:10.48550/arxiv.1703.00810 · Cited by 1,674 · cited 2× · see citing works
“In [29], they extend the approach of [28] demonstrating the effectiveness of the Information Plane visualization of DNNs.”
BackgroundFederated Learning for Mobile Keyboard Prediction
Hard, A., Rao, K., Mathews, R., Ramaswamy, S., Beaufays, F., Augenstein, S., Eichner, H., Kiddon, C., & Ramage, D. (2018) · arXiv · doi:10.48550/arxiv.1811.03604 · Cited by 1,148 · cited 1× · see citing works
“Federated learning plays a critical role in various privacy-sensitive scenarios, such as power intelligence applications like keyboard prediction and emoji prediction in smart phone [9,10], optimising patient’s healthcare in hospitals [11], helping internet of things (IoT) systems adapt to environmental changes [12] e…”
BackgroundEdge-centric Computing: Vision and Challenges
Garcia Lopez, P., Montresor, A., Epema, D., Datta, A., Higashino, T., Iamnitchi, A., Barcellos, M., Felber, P., & Riviere, E. (2015) · ACM SIGCOMM Computer Communication Review · doi:10.1145/2831347.2831354 · Cited by 934 · see citing works
Mutual Information between Discrete and Continuous Data Sets
Ross, B. C. (2014) · PLoS ONE · doi:10.1371/journal.pone.0087357 · Cited by 869 · cited 1× · see citing works
“MI can detect any kind of relationship between random variables even those that do not manifest themselves in the covariance [21], and has a straightforward interpretation as the amount of shared information between datasets (measured in, for example, bits) [22].”
BackgroundLearning Differentially Private Recurrent Language Models
McMahan, H. B., Ramage, D., Talwar, K., & Zhang, L. (2017) · arXiv · doi:10.48550/arxiv.1710.06963 · Cited by 849 · see citing works
Haghi, M., Thurow, K., & Stoll, R. (2017) · Healthcare Informatics Research · doi:10.4258/hir.2017.23.1.4 · Cited by 712 · cited 1× · see citing works
“Nowadays, more and more intelligent devices, such as smart phones, wearable devices and autonomous vehicles, are widely used [1,2], generating a wealth of data.”
BackgroundACM SIGCOMM computer communication review
Campbell, A. T., & Schwartz, M. (2001) · ACM SIGCOMM Computer Communication Review · doi:10.1145/1037107.1037111 · Cited by 600 · see citing works
On the information bottleneck theory of deep learning*
Saxe, A., Bansal, Y., Dapello, J., Advani, M., Kolchinsky, A., Tracey, B., & Cox, D. (2019) · Journal of Statistical Mechanics Theory and Experiment · doi:10.1088/1742-5468/ab3985 · Cited by 516 · cited 1× · see citing works
“These results are later questioned by [30], who argue that the compression phase is not general in the DNN training process.”
BackgroundProceedings of the 2020 ACM SIGSAC Conference on Computer and Communications Security
Nafees, M. N., Saxena, N., Burnap, P., & Choi, B. J. (2020) · doi:10.1145/3372297 · Cited by 356 · see citing works
2020 IEEE Global Communications Conference (GLOBECOM)
(2020) · Globecom · doi:10.1109/globecom42002.2020.9322132 · Cited by 333 · see citing works
Federated Learning for Emoji Prediction in a Mobile Keyboard
Ramaswamy, S., Mathews, R., Rao, K., & Beaufays, F. (2019) · arXiv · doi:10.48550/arxiv.1906.04329 · Cited by 231 · cited 2× · see citing works
“Federated learning plays a critical role in various privacy-sensitive scenarios, such as power 30 intelligence applications like keyboard prediction and emoji prediction in smart phone [9] [10], 31 optimising patient’s healthcare in hospitals[11], helping internet of things (IoT) systems adapt to 32 environmental chan…”
BackgroundLi, H., Yin, Y., Fu, Z., Zhang, S., Deng, H., & Liu, D. (2020) · PLoS ONE · doi:10.1371/journal.pone.0230706 · Cited by 212 · see citing works
Modern Mathematical Methods for Physicists and Engineers
Cantrell, C. D. (2000) · Cambridge University Press eBooks · doi:10.1017/9780511811487 · Cited by 204 · see citing works
Proceedings of the first edition of the MCC workshop on Mobile cloud computing
Gerla, M., Finland, S. '. A. S. 2. C. 2. H., Association for Computing Machinery-Digital Library, ACM Special Interest Group on Data Communication, & SIGs, A. (2012) · Cited by 202 · see citing works
Nonlinear Information Bottleneck
Kolchinsky, A., Tracey, B., & Wolpert, D. H. (2019) · Entropy · doi:10.3390/e21121181 · Cited by 168 · cited 1× · see citing works
“Following this line of work, [32] propose a method for performing IP on arbitrarily-distributed discrete and/or continuous variables, while [33] use matrix-based R ´ enyi (cid:48) s entropy to provide comprehensive IP analysis of large DNNs.”
MethodComputer Graphics and Image Processing.
Freeman, H. (1976) · Defense Technical Information Center (DTIC) · Cited by 141 · see citing works
Graph Oracle Models, Lower Bounds, and Gaps for Parallel Stochastic Optimization
Woodworth, B. E., Wang, J.-L., Smith, A. D., McMahan, H. B., & Srebro, N. (2018) · Neural Information Processing Systems · doi:10.48550/arxiv.1805.10222 · Cited by 132 · cited 1× · see citing works
“In [18], they suggest a general oracle-based framework in parallel stochastic optimization.”
BackgroundCOLA: Decentralized Linear LearningPreprint
He, L., Bian, A., & Jaggi, M. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1808.04883 · Cited by 94 · see citing works
Scalable Mutual Information Estimation Using Dependence Graphs
Noshad, M., Zeng, Y., & Hero, A. O. (2019) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2019.8683351 · Cited by 76 · see citing works
The world in 2025 - predictions for the next ten years
Taylor, R., Baron, D. A., & Schmidt, D. J. (2015) · International Microsystems, Packaging, Assembly and Circuits Technology Conference · doi:10.1109/impact.2015.7365193 · Cited by 73 · cited 1× · see citing works
“With the increasing amount of generated data and increasing computing power of the smart devices [3], recent studies have explored distributed training of models at these edging devices [4,5].”
BackgroundSemi-Cyclic Stochastic Gradient Descent
Eichner, H., Koren, T., McMahan, H. B., Srebro, N., & Talwar, K. (2019) · arXiv · doi:10.48550/arxiv.1904.10120 · Cited by 66 · cited 1× · see citing works
“In [19], they investigate a pluralistic solution to address the cyclic patterns in data samples during federated training.”
BackgroundLoAdaBoost:Loss-Based AdaBoost Federated Machine Learning on medical DataPreprint
Li, H., Yin, Y., Fu, Z., Zhang, S., Deng, H., & Liu, D. (2018) · arXiv (Cornell University) · Cited by 57 · see citing works
Federated Learning for Ultra-Reliable Low-Latency V2V Communications
Samarakoon, S., Bennis, M., Saad, W., & Debbah, M. (2018) · 2018 IEEE Global Communications Conference (GLOBECOM) · doi:10.1109/glocom.2018.8647927 · Cited by 25 · see citing works
Wickstrøm, K., Løkse, S., Kampffmeyer, M., Yu, S., Príncipe, J. C., & Jenssen, R. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.11396 · Cited by 23 · cited 1× · see citing works
“Following this line of work, [32] propose a method for performing IP on arbitrarily-distributed discrete and/or continuous variables, while [33] use matrix-based Rényi′s entropy to provide comprehensive IP analysis of large DNNs.”
MethodLi, H., Yin, Y., Fu, Z., et al. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1811.12629 · Cited by 13 · cited 1× · see citing works
“Federated learning plays a critical role in various privacy-sensitive scenarios, such as power 30 intelligence applications like keyboard prediction and emoji prediction in smart phone [9] [10], 31 optimising patient’s healthcare in hospitals[11], helping internet of things (IoT) systems adapt to 32 environmental chan…”
BackgroundInformation-Theoretic Perspective of Federated LearningPreprint
Adilova, L., Rosenzweig, J., & Kamp, M. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1911.07652 · Cited by 1 · cited 3× · see citing works
“In contrast to [34] and the other work reviewed above, in this paper, we treat each local model as a continuous random vector, and directly measure the MI between different local models through the training phase using two methods.”
BackgroundMethodInfluential
Flagged first, then by citations
5 of the 47 references couldn’t be matched to a known paper.
Research works printed but not linked: 3
- [7]McMahan H.B. Moore E. Ramage D. Hampson S. Communication-efficient learning of deep networks from decentralized data arXiv 2016 1602.05629
- [16]Agarwal N. Suresh A.T. Yu F.X.X. Kumar S. McMahan B. cpSGD: Communication-efficient and differentially- private distributed SGD Proceedings of the Advances in Neural Information Processing Systems Montreal, QC, Canada 3–8 December 2018 7564 7575
- [37]Kingma D.P. Ba J. Adam: A method for stochastic optimization arXiv 2014 1412.6980
Cited by
Sorted by citations
A review of applications in federated learning
Li, L., Fan, Y., Tse, M., & Lin, K.-Y. (2020) · Computers & Industrial Engineering · doi:10.1016/j.cie.2020.106854 · Cited by 1,559 · see citing works
“Xiao et al., (2020) detect that the mutual information is increased which implies correlation between clients, while the distance of parameters is getting greater with increased iteration.”
BackgroundTowards Personalized Federated Learning
Tan, A. Z., Yu, H., Cui, L., & Yang, Q. (2022) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2022.3160699 · Cited by 1,366 · see citing works
Blockchain-empowered Federated Learning: Challenges, Solutions, and Future Directions
Zhu, J., Cao, J., Saxena, D., Jiang, S., & Ferradi, H. (2023) · ACM Computing Surveys · doi:10.1145/3570953 · Cited by 309 · see citing works
“The ideas have recently been extended to federated learning [16, 38, 108].”
BackgroundSurvey: federated learning data security and privacy-preserving in edge-Internet of Things
Li, H., Ge, L., & Tian, L. (2024) · Artificial Intelligence Review · doi:10.1007/s10462-024-10774-7 · Cited by 112 · see citing works
“Decentralized training has also been shown to be more efficient than centralized training when running federated systems in low-bandwidth or high-latency networks (Xiao et al. 2020; Liu et al. 2020; Jiang et al. 2020).”
FedMed: A Federated Learning Framework for Language Modeling
Wu, X., Liang, Z., & Wang, J. (2020) · Sensors · doi:10.3390/s20144048 · Cited by 67 · see citing works
“[25] calculated mutual information of two worker models and suggest the averaging method is not the optimal way based on entropy theory.”
MethodLightweight Blockchain-Empowered Secure and Efficient Federated Edge Learning
Jin, R. M., Hu, J. W., Min, G., & Mills, J. (2023) · IEEE Transactions on Computers · doi:10.1109/tc.2023.3293731 · Cited by 62 · see citing works
“As shown in [32], the MI between honest clients shows an increasing trend with training round, instead of the Euclidean distance.”
Ge, L., Li, H., Wang, X., & Wang, Z. (2023) · Neurocomputing · doi:10.1016/j.neucom.2023.126897 · Cited by 54 · see citing works
Uncovering the Potential of Indoor Localization: Role of Deep and Transfer Learning
Kerdjidj, O., Himeur, Y., Sohail, S. S., Amira, A., Fadli, F., Atalla, S., Mansoor, W., Copiaco, A., Daradkeh, M. K., Gawanmeh, A., et al. (2024) · IEEE Access · doi:10.1109/access.2024.3402997 · Cited by 53 · see citing works
“Some related works using this technique can be cited here [171], [172], [173].”
MethodNguyen, N. H., Nguyen, P. L., Nguyen, T., Nguyen, T.-H., Nguyen, D. L., Nguyen, T.-H., Pham, H., & Truong, T. N. (2022) · Proceedings of the International Conference on Parallel Processing · doi:10.1145/3545008.3545085 · Cited by 37 · see citing works
Individualised Responsible Artificial Intelligence for Home-Based Rehabilitation
Vourganas, I., Stanković, V., & Stanković, L. (2020) · Sensors · doi:10.3390/s21010002 · Cited by 37 · see citing works
“Moreover, for some applications which require an increased level of privacy and security, federated learning could also be suggested [35].”
BackgroundFedCav: Contribution-aware Model Aggregation on Distributed Heterogeneous Data in Federated Learning
Zeng, H. Y., Zhou, T., Guo, Y., Cai, Z., & Liu, F. (2021) · Proceedings of the International Conference on Parallel Processing · doi:10.1145/3472456.3472504 · Cited by 27 · see citing works
“This will accidentally increase the time for the training to reach convergence and also reduce the performance on model accuracy [21].”
BackgroundAn Efficient and Robust Aggregation Algorithm for Learning Federated CNN
Lu, Y., & Fan, L. (2020) · doi:10.1145/3432291.3432303 · Cited by 26 · see citing works
“However, Xiao et al.[11] pointed that Average may not be an optimum way of aggregating.”
BackgroundA Systematic Literature Review on Federated Learning: From A Model Quality Perspective
Liu, Y., Zhang, L., Ge, N., & Li, G. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2012.01973 · Cited by 36 · see citing works
“RNN (Recurrent Neural Network) H, V 24 [2, 4, 24,27,30,32–35,38,44,62, 65,70,88,93,96,101,102,105,108, 115,118,119]”
Probabilistic Predictions with Federated Learning
Thorgeirsson, A. T., & Gauterin, F. (2020) · Entropy · doi:10.3390/e23010041 · Cited by 24 · see citing works
“showed that during FL, client weights become increasingly correlated but not closer to each other in terms of distance metrics [39].”
BackgroundHow to Prevent the Poor Performance Clients for Personalized Federated Learning?
Qu, Z., Li, X., HAN, X. J., Duan, R., Shen, C., & Chen, L. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01171 · Cited by 20 · see citing works
“Though many pFL algorithms make accomplishments by modifying the universal learning process [41, 47] or enhancing the personalization [3, 27, 37], they may lead part of clients to fall into poor learning performance, where the personalization of local clients performs a large statistical deviation from the “averaged d…”
BackgroundUncertainty-Aware Aggregation for Federated Open Set Domain Adaptation
Qin, Z., Liu, Y., Gao, F., Hu, Q., & Shen, C. (2022) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2022.3214930 · Cited by 19 · see citing works
Jeong, M., Choi, S., & Kim, S. (2023) · ICT express · doi:10.1016/j.icte.2023.01.009 · Cited by 15 · see citing works
Agrawal, V., Kalmady, S. V., Malipeddi, V. M., Manthena, M. V., Sun, W., Islam, M. S., Hindle, A., Kaul, P., & Greiner, R. (2024) · International Conference on Medical and Health Informatics (ICMHI) · doi:10.1145/3673971.3673990 · Cited by 16 · see citing works
Nguyen, N. H., Nguyen, D. L., Nguyen, T. B., Nguyen, T., Pham, H.-H., Nguyen, T., & Nguyen, P. L. (2023) · IEEE/ACM International Symposium on Cluster, Cloud and Internet Computing · doi:10.1109/ccgrid57682.2023.00032 · Cited by 15 · see citing works
Przybyła‐Kasperek, M., & Marfo, K. F. (2021) · Entropy · doi:10.3390/e23121568 · Cited by 15 · see citing works
“In the literature, we can find applications of neural networks in the stage of predictions fusion [9–11].”
MethodFedns: Improving Federated Learning for Collaborative Image Classification on Mobile Clients
Zhuo, Y., & Li, B. (2021) · IEEE International Conference on Multimedia and Expo (ICME) · doi:10.1109/icme51207.2021.9428075 · Cited by 14 · see citing works
“Second, simple weighted average of client models may not be optimal, as analyzed in some recent work [2].”
BackgroundConcepts, Key Challenges and Open Problems of Federated Learning
Iqbal, Z., & Chan, H. Y. (2021) · International Journal of Engineering · doi:10.5829/ije.2021.34.07a.11 · Cited by 14 · see citing works
Li, Y., Xu, G., Meng, X., Du, W., & Ren, X. (2024) · Entropy · doi:10.3390/e26050353 · Cited by 12 · see citing works
“Some studies show that averaging parameters may not be the optimum way of aggregating trained parameters [4].”
BackgroundLabel-Aware Aggregation for Improved Federated Learning
Khalil, A., Wainakh, A., Zimmer, E., Parra‐Arnau, J., Anta, A. F., Meuser, T., & Steinmetz, R. (2023) · International Conference on Fog and Mobile Edge Computing (FMEC) · doi:10.1109/fmec59375.2023.10306055 · Cited by 10 · see citing works
“Xiao et al. [18] studied the behavior of the model parameters of different clients and demonstrated that, with the increase of training iterations, the model parameters of different clients become more correlated but not necessarily closer in value.”
BackgroundByzantine-Robust and Efficient Federated Learning for the Internet of Things
Jin, R. M., Hu, J. W., Min, G., & Lin, H. (2022) · IEEE Internet of Things Magazine · doi:10.1109/iotm.001.2100192 · Cited by 10 · see citing works
Ghosh, A., & Krishnamoorthy, P. (2024) · IEEE Access · doi:10.1109/access.2024.3462171 · Cited by 8 · see citing works
“FedAvg [1], [20], [21], [34], [35] is the foundational approach in federated learning, where local models are trained independently on each client’s data and then aggregated to form a global model.”
BackgroundPham, Q. H., Nguyen, N. H., Nguyen, T., Pham, H.-H., Nguyen, P. L., & Nguyen, T. T. (2023) · International Symposium on Computing and Networking - Across Practical Development and Theoretical Research · doi:10.1109/candar60563.2023.00023 · Cited by 6 · see citing works
“…regard non-IID data as a context of (i) quantity skew non-IID , i.e., different clients hold different amounts of data, e.g., [2], [6], [10], [17] or (ii) label skew non-IID , i.e., the label distribution differs between clients such as by distributing samples of a particular label 1 to the…”
BackgroundSong, Y., Ai, Y., Xiao, X., Liu, Z., Tang, Z., & Li, K. (2024) · Journal of Systems Architecture · doi:10.1016/j.sysarc.2024.103070 · Cited by 6 · see citing works
SplitMAC: Wireless Split Learning Over Multiple Access Channels
Kim, S., Oh, Y., & Jeon, Y. (2024) · IEEE Transactions on Wireless Communications · doi:10.1109/twc.2024.3486377 · Cited by 6 · see citing works
“This phenomenon has been reported in [28], [29] by demonstrating that the heterogeneity of datasets and simple averaging degrade the performance of distributed learning.”
BackgroundDuc, M. V., Dang, P. M., Phuong, T. T., Truong, T. D., Vu, H., & Thanh, N. H. (2025) · Future Internet · doi:10.3390/fi17070299 · Cited by 5 · see citing works
“Model parameters do not always accurately reflect the underlying characteristics of the input data [16], especially when local training is minimal.”
A robust federated biased learning algorithm for time series forecasting
Song, M., Zhao, X., & Pedrycz, W. (2025) · Cluster Computing · doi:10.1007/s10586-024-05061-7 · Cited by 5 · see citing works
“As peng et al. [20] put forward, averaging parameters may not be the optimum way of aggregating trained parameters.”
FedUA: An Uncertainty-Aware Distillation-Based Federated Learning Scheme for Image Classification
Lee, S.-M., & Wu, J. (2023) · Information · doi:10.3390/info14040234 · Cited by 5 · see citing works
Zeng, H., Zhou, T., Guo, Y., Cai, Z., & Liu, F. (2024) · Journal of Parallel and Distributed Computing · doi:10.1016/j.jpdc.2024.104994 · Cited by 4 · see citing works
Entropy-Driven Stochastic Federated Learning in Non-IID 6G Edge-RAN
Aamer, B., Chergui, H., Benjillali, M., & Verikoukis, C. V. (2021) · Frontiers in Communications and Networks · doi:10.3389/frcmn.2021.739414 · Cited by 4 · see citing works
“In this context, it has been con fi rmed that the correlation between the model parameters of different clients is increasing as the training progresses, which implies that aggregating parameters directly by averaging may not be a reasonable approach in general (Xiao et al., 2020).”
BackgroundEntropy-Driven Stochastic Policy for Fast Federated Learning in Beyond 5G Edge-RAN
Aamer, B., Chergui, H., Benjillali, M., & Verikoukis, C. V. (2021) · Globecom · doi:10.1109/globecom46510.2021.9685786 · Cited by 4 · see citing works
“In this context, it has been confirmed that the correlation between the model parameters of different clients is increasing as the training progresses, which implies that aggregating parameters directly by averaging may not be a reasonable approach in general [9].”
BackgroundSAAFL: Secure Aggregation for Label-Aware Federated Learning
Akram, A., Pham, H. N. H., Önen, M., & Gritti, C. (2025) · IFIP advances in information and communication technology · doi:10.1007/978-3-031-92882-6_3 · Cited by 3 · see citing works
“Unfortunately, recent findings have shown that FedAvg falls short in achieving good model accuracy level when datasets are heterogeneous or non-identically distributed (non-IID) [15, 19].”
Federated Learning Technology in Serial Topology for IoT Networks
Sheng, J., Xiong, J., & Liu, B. (2023) · IEEE International Symposium on Broadband Multimedia Systems and Broadcasting (BMSB) · doi:10.1109/bmsb58369.2023.10211485 · Cited by 3 · see citing works
F2NAS: Flexible Federated Neural Architecture Search in Green Edge Computing
Zhao, Z., Qiu, C., Zhao, Y., Wang, X., Yao, H., Li, X., & Yu, F. R. (2024) · International Conference on Communications · doi:10.1109/icc51166.2024.10623100 · Cited by 3 · see citing works
“Several distance computation methods, such as Euclidean distance, Mahalanobis distance, etc., have demonstrated suitability in measuring model similarity [13]–[15].”
MethodRobust Model Aggregation for Federated Learning with Heterogeneous Clients
Xu, R., Feng, X., & Zheng, H. (2021) · IEEE International Conference on Computer and Communications (ICCC) · doi:10.1109/iccc54389.2021.9674541 · Cited by 3 · see citing works
Bayesian Federated Neural Matching That Completes Full Information
Xiao, P., & Cheng, S. S. M. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i9.26245 · Cited by 2 · see citing works
Self-citationEnergy-Friendly Federated Neural Architecture Search for Industrial Cyber-Physical Systems
Wang, X., Zhao, Y., Qiu, C., Gao, F., Zhao, Z., Yao, H., & Li, X. (2025) · IEEE Journal on Selected Areas in Communications · doi:10.1109/jsac.2025.3574599 · Cited by 1 · see citing works
“Various distance calculation methods, such as the Euclidean distance and the Mahalanobis distance, can be used to measure model similarity [41], [42], [43].”
Overview of Federated Learning and Its Advantages
Kakkar, A., & Kumar, S. (2024) · Advances in healthcare information systems and administration book series · doi:10.4018/979-8-3693-1874-4.ch016 · Cited by 1 · see citing works
A Secure and Communication‐Efficient Federated Recommendation Method With Blockchain
Lu, S., Huang, D., Wang, Z., Li, Z., Zhang, H., Dou, W., & Tian, C. (2025) · Concurrency and Computation Practice and Experience · doi:10.1002/cpe.70219 · Cited by 1 · see citing works
“As noted by prior research [51], mutual information (MI) between honest clients increases over time, unlike Euclidean distance measures.”
Probabilistic Federated Learning of Neural Networks Incorporated with Global Posterior InformationPreprint
Xiao, P., & Cheng, S. S. M. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2012.03178 · Cited by 1 · see citing works
“However, this coordinate-wise averaging in FedAvg was shown to be not-optimum [21] and the performance of FedAvge can also deteriorate significantly as statistical heterogeneity in non-i.i.d (Independent and Identically Distributed) data increases [10].”
BackgroundSelf-citationZhang, Z., Zhou, M., Niu, K., & Abdallah, C. T. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2108.03508 · Cited by 1 · see citing works
“Some researchers have also discovered that models could grow further apart (measured by mutual information) while still increasing their parameters’ correlations, thus showing us a different explanation to why the whole model distances are growing with each epoch [Xiao et al., 2020].”
BackgroundFederated Learning with Random Communication and Dynamic Aggregation
Huang, R., Lu, T., Luo, Y., Liu, G., & Chang, S. (2021) · International Conference on Machine Learning Techniques and Data Science (MLDS) · doi:10.5121/csit.2021.111816
Personalized Federated Aggregation Algorithm based on Local Attention Mechanism
Zeng, Y., Yang, Y., Yao, T., & He, W. (2023) · International Symposium on Parallel Architectures, Algorithms and Programming · doi:10.1109/paap60200.2023.10391300
“Although many studies have extended upon the FedAvg algorithm since its introduction, FedAvg itself has notable drawbacks[19].”
BackgroundLai, J., Yang, Y., Wu, H., Mai, X., Li, Z., & Mei, P. (2023) · International Conference on Digital Society and Intelligent Systems (DSInS) · doi:10.1109/dsins60115.2023.10455553
“However, it has no convergence guarantee for non-convex problems and is difficult to converge on Non-IID datasets [3], showing certain defects [4].”
BackgroundFederated Learning for Cooperative Heterogeneous Agents
Saadati, Y., & Amini, M. H. (2024) · Springer eBooks · doi:10.1007/978-1-4614-6624-6_91-1
Optimization Analysis of Power Grid Data Quality by Federal Average Algorithm
Tang, L., Li, B., Zeng, T., Liu, H., & Shen, Y. (2025) · International Journal of High Speed Electronics and Systems · doi:10.1142/s0129156425402840
AdaCons: Adaptive Consensus Gradient Aggregation for Distributed Training
Choukroun, Y., Azoulay, S., & Kisilev, P. (2026) · Lecture notes in networks and systems · doi:10.1007/978-3-032-04160-9_3
FedVSR: Towards Model-Agnostic Federated Learning in Video Super-Resolution
Dehaghi, A. M., KhademSohi, H., Razavi, R., Drew, S., & Moshirpour, M. (2026) · ACM Multimedia Systems Conference (MMSys) · doi:10.1145/3793853.3795752
- Authors
- Xiao, P., Cheng, S., Stankovic, V., & Vukobratovic, D.
- Affiliations
- Tongji University (CN)
- Department of Electronic and Electrical engineering, University of Strathclyde, Glasgow G1 1XW, UK
- Faculty of Technical Sciences, University of Novi Sad, 21000 Novi Sad, Serbia
- Corresponding author
- Cheng, Samuel
- Venue
- Entropy
- Publisher
- MDPI AG
- Published
- 2020-03-11
- Volume
- 22
- Issue
- 3
- Pages
- 314
- ISSN
- 1099-4300 (ISSN-L)
- PMC
- PMC7516771
- OpenAlex
- W2999593354
- Cited by
- 54
- References
- 47
- Citations by year
- 2026: 2 · 2025: 6 · 2024: 11 · 2023: 12 · 2022: 5 · 2021: 10 · 2020: 8
- FWCI
- 3.44
- Type
- journal article
- Language
- en
- Open access
- gold
- dx.doi.org
- Licence
- Locations
- Entropy (publishedVersion, cc-by)
- Preprints.org (acceptedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Strathprints: The University of Strathclyde institutional repository (University of Strathclyde) (acceptedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- European Commission
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Privacy-Preserving Technologies in Data
- Physical Sciences › Computer Science › Artificial Intelligence › Stochastic Gradient Optimization Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Keywords
- federated learning, parameter aggregation, averaging, mutual information, stochastic gradient descent, data privacy, decentralized learning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
