Federated Gradient Averaging for Multi-Site Training with Momentum-Based Optimizers
Open accessAbstract
Multi-site training methods for artificial neural networks are of particular interest to the medical machine learning community primarily due to the difficulty of data sharing between institutions. However, contemporary multi-site techniques such as weight averaging and cyclic weight transfer make theoretical sacrifices to simplify implementation. In this paper, we implement federated gradient averaging (FGA), a variant of federated learning without data transfer that is mathematically equivalent to single site training with centralized data. We evaluate two scenarios: a simulated multi-site dataset for handwritten digit classification with MNIST and a real multi-site dataset with head CT hemorrhage segmentation. We compare federated gradient averaging to single site training, federated weight averaging (FWA), and cyclic weight transfer. In the MNIST task, we show that training with FGA results in a weight set equivalent to centralized single site training. In the hemorrhage segmentation task, we show that FGA achieves on average superior results to both FWA and cyclic weight transfer due to its ability to leverage momentum-based optimization.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Federated learning has shown promise in recent medical imaging applications, including COVID-19 analysis.
“FL has already shown promise in recent medical imaging applications40–43, including in COVID-19 analysis8,44,45.”
From Federated learning for predicting clinical outcomes in patients with COVID-19 (2021) · cites this paper as [42]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
U-Net: Convolutional Networks for Biomedical Image Segmentation
Ronneberger, O., Fischer, P., & Brox, T. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-24574-4_28 · Cited by 99,404 · see citing works
Pan, S. J., & Yang, Q. (2010) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2009.191 · Cited by 23,387 · see citing works
Communication-Efficient Learning of Deep Networks from Decentralized Data
McMahan, H. B., Moore, E., Ramage, D., Hampson, S., & Arcas, B. A. Y. (2016) · arXiv · doi:10.48550/arxiv.1602.05629 · Cited by 14,215 · cited 6× · see citing works
“Although recent works [15] have shown the theoretical convergence for federated weight averaging (FWA, previously formulated as FedAVG [19]), in practice there is still a performance gap between centralized training at a single site and collaborative learning methods [28].”
BackgroundMethodInfluentialOvercoming catastrophic forgetting in neural networks
Kirkpatrick, J., Pascanu, R., Rabinowitz, N., Veness, J., Desjardins, G., Rusu, A. A., Milan, K., Quan, J., Ramalho, T., Grabska-Barwinska, A., et al. (2017) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1611835114 · Cited by 7,899 · see citing works
How transferable are features in deep neural networks?
Yosinski, J., Clune, J., Bengio, Y., & Lipson, H. (2014) · arXiv · doi:10.48550/arxiv.1411.1792 · Cited by 6,061 · cited 1× · see citing works
“In transfer learning [22, 34], also known as pre-training or finetuning, model parameters are learnt first on some (usually large) dataset and subsequently trained for the problem of interest on a smaller dataset.”
BackgroundMembership Inference Attacks Against Machine Learning Models
Shokri, R., Stronati, M., Song, C., & Shmatikov, V. (2017) · 2017 IEEE Symposium on Security and Privacy (SP) · doi:10.1109/sp.2017.41 · Cited by 4,754 · cited 2× · see citing works
“However, the reconstruction of training data from the weights or gradients alone is nontrivial [30].”
BackgroundPrivacy-preserving deep learning
Shokri, R., & Shmatikov, V. (2015) · Proceedings of the 22nd ACM SIGSAC Conference on Computer and Communications Security · doi:10.1145/2810103.2813687 · Cited by 2,356 · cited 1× · see citing works
“Previous works have demonstrated the dangers of sending raw data through the network [30] and to this end differentially private (DP) methods have been proposed [8, 29, 14].”
BackgroundCatastrophic forgetting in connectionist networks
French, R. (1999) · Trends in Cognitive Sciences · doi:10.1016/s1364-6613(99)01294-2 · Cited by 2,311 · see citing works
Practical Recommendations for Gradient-Based Training of Deep Architectures
Bengio, Y. (2012) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-35289-8_26 · Cited by 2,259 · cited 1× · see citing works
“Gradient averaging has been a standard practice since the advent of deep learning [3] with regards to training a neural network using batches, as weight updates from a single sample are noisy and updates from the entire dataset are undesirable for empirical and theoretical reasons.”
BackgroundSheller, M. J., Edwards, B., Reina, G. A., Martin, J., Pati, S., Kotrotsou, A., Milchenko, M., Xu, W., Marcus, D., Colen, R. R., et al. (2020) · Scientific Reports · doi:10.1038/s41598-020-69250-1 · Cited by 1,646 · see citing works
On the Convergence of FedAvg on Non-IID Data
Li, X., Huang, K., Yang, W., Wang, S., & Zhang, Z. (2019) · arXiv · doi:10.48550/arxiv.1907.02189 · Cited by 1,575 · cited 1× · see citing works
“Although recent works [15] have shown the theoretical convergence for federated weight averaging (FWA, previously formulated as FedAVG [19]), in practice there is still a performance gap between centralized training at a single site and collaborative learning methods [28].”
BackgroundFederated Optimization: Distributed Machine Learning for On-Device Intelligence
Konečný, J., McMahan, H. B., Ramage, D., & Richtárik, P. (2016) · arXiv · doi:10.48550/arxiv.1610.02527 · Cited by 1,563 · cited 1× · see citing works
“The term “federated learning” (FL) was first widely known from [12] in the application of learning from mobile devices.”
BackgroundDeep Models Under the GAN: Information Leakage from Collaborative Deep Learning
Hitaj, B., Ateniese, G., & Perez-Cruz, F. (2017) · Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security · doi:10.1145/3133956.3134012 · Cited by 1,377 · see citing works
QSGD: Communication-Efficient SGD via Gradient Quantization and Encoding
Alistarh, D., Grubic, D., Li, J. Z., Tomioka, R., & Vojnović, M. (2016) · London School of Economics and Political Science Research Online (London School of Economics and Political Science) · doi:10.48550/arxiv.1610.02132 · Cited by 1,189 · see citing works
Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Training
Lin, Y., Han, S., Mao, H., Wang, Y., & Dally, W. J. (2017) · arXiv · doi:10.48550/arxiv.1712.01887 · Cited by 1,134 · cited 1× · see citing works
“There is also recent research to transform the gradients into communication-efficient variants [32, 33, 2, 16].”
BackgroundTernGrad: Ternary Gradients to Reduce Communication in Distributed Deep LearningPreprint
Wen, W., Xu, C., Yan, F., Wu, C., Wang, Y., Chen, Y., & Li, H. H. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1705.07878 · Cited by 718 · cited 2× · see citing works
“There is also recent research to transform the gradients into communication-efficient variants [32, 33, 2, 16].”
BackgroundMethodYu, H., Yang, S., & Zhu, S.-H. (2018) · AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33015693 · Cited by 696 · cited 1× · see citing works
“Additionally, many methods have considered gradient averaging as incurring too much network overhead, which is one reason why weight averaging is preferred [19, 35].”
BackgroundPrivacy-Preserving Federated Brain Tumour Segmentation
Li, W., Milletarì, F., Xu, D., Rieke, N., Hancox, J., Zhu, W., Baust, M., Cheng, Y., Ourselin, S., Cardoso, M. J., et al. (2019) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-32692-0_16 · Cited by 578 · see citing works
Variational Continual Learning
Nguyen, C. V., Li, Y., Bui, T. D., & Turner, R. E. (2017) · arXiv · doi:10.48550/arxiv.1710.10628 · Cited by 491 · cited 1× · see citing works
“Related research in the field of continual learning [11, 20, 20] investigates methods to mitigate catastrophic forgetting [6], the phenomenon by which neural networks have no guarantee to retain useful weight values when training on new data.”
BackgroundDistributed deep learning networks among institutions for medical imaging
Chang, K., Balachandar, N., Lam, C., Yi, D., Brown, J., Beers, A., Rosen, B., Rubin, D. L., & Kalpathy-Cramer, J. (2018) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocy017 · Cited by 340 · see citing works
Health Insurance Portability and Accountability Act
Wiley, D. C., & Cory, A. C. (2013) · Encyclopedia of School Health · doi:10.4135/9781452276250.n118 · Cited by 236 · see citing works
Health Insurance Portability and Accountability Act of 1996. Public Law 104-191.
(1996) · PubMed · Cited by 213 · see citing works
mHealth Data Security: The Need for HIPAA-Compliant Standardization
Luxton, D. D., Kayl, R. A., & Mishkind, M. C. (2012) · Telemedicine Journal and e-Health · doi:10.1089/tmj.2011.0180 · Cited by 169 · see citing works
Communication-Computation Efficient Gradient CodingPreprint
Ye, M., & Abbe, E. A. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1802.03475 · Cited by 126 · cited 1× · see citing works
“There is also recent research to transform the gradients into communication-efficient variants [32, 33, 2, 16].”
BackgroundAsynchronous stochastic gradient descent for DNN training
Zhang, S., Zhang, C., You, Z., Zheng, R., & Xu, B. (2013) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2013.6638950 · Cited by 125 · cited 2× · see citing works
“Similarly, asynchronous stochastic gradient descent [36] considered different GPU devices as clients, splitting training data among them and aggregating gradients before performing weight update to decrease training time.”
BackgroundMethodProtected health information on social networking sites: ethical and legal considerations
Thompson, L. A., Black, E., Duff, W. P., Paradise Black, N., Saliba, H., & Dawson, K. (2011) · Journal of Medical Internet Research · doi:10.2196/jmir.1590 · Cited by 114 · see citing works
Inter-site Variability in Prostate Segmentation Accuracy Using Deep Learning
Gibson, E., Hu, Y., Ghavami, N., Ahmed, H. U., Moore, C. M., Emberton, M., Huisman, H. J., & Barratt, D. C. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-00937-3_58 · Cited by 66 · cited 1× · see citing works
“site’s data perform poorly on a new site [7].”
BackgroundAuthor Correction: Non-professional phagocytosis: a general feature of normal tissue cells
Seeberg, J. C., Loibl, M., Moser, F., Schwegler, M., Büttner-Herold, M. J., Daniel, C., Engel, F. B., Hartmann, A., Schlötzer-Schrehardt, U. M., Goppelt‐Struebe, M., et al. (2020) · Scientific Reports · doi:10.1038/s41598-020-65963-5 · Cited by 38 · see citing works
Distributed deep learning across multisite datasets for generalized CT hemorrhage segmentation
Remedios, S. W., Roy, S., Bermudez, C., Patel, M. B., Butman, J. A., Landman, B. A., & Pham, D. L. (2020) · Medical Physics · doi:10.1002/mp.13880 · Cited by 37 · see citing works
Self-citationMultiple Network Disconnection in Anosognosia for Hemiplegia
Monai, E., Bernocchi, F., Bisio, M., Bisogno, A. L., Salvalaggio, A., & Corbetta, M. (2020) · Frontiers in Systems Neuroscience · doi:10.3389/fnsys.2020.00021 · Cited by 33 · see citing works
The HIPAA Privacy Rule and Protected Health Information
Fetzer, D. T., & West, O. C. (2008) · Academic Radiology · doi:10.1016/j.acra.2007.11.008 · Cited by 32 · see citing works
Distributed Weight Consolidation: A Brain Segmentation Case Study
McClure, P., Zheng, C. Y., Kaczmarzyk, J., Lee, J. A., Ghosh, S., Nielson, D. M., Bandettini, P. A., & Pereira, F. (2018) · PubMed · doi:10.48550/arxiv.1805.10863 · Cited by 19 · cited 1× · see citing works
“Bayesian neural networks, in which parameters are not point estimates but distributions of parameters, allow techniques such as distributed weight consolidation [18] to consider multi-site training as a continual learning problem.”
Background
Flagged first, then by citations
1 of the 33 references couldn’t be matched to a known paper.
Research works printed but not linked: 3
- [8]Goodfellow I, Bengio Y, Courville A: Deep learning. MIT press (2016)
- [10]Kingma DP, Ba J: Adam: A method for stochastic optimization. arXiv preprintarXiv:1412.6980 (2014)
- [27]Ruder S: An overview of gradient descent optimization algorithms. arXiv preprintarXiv:1609.04747 (2016)
Cited by
Sorted by citations
Federated learning for predicting clinical outcomes in patients with COVID-19
Dayan, I., Roth, H. R., Zhong, A., Harouni, A., Gentili, A., Abidin, A. Z., Liu, A., Costa, A. B., Wood, B. J., Tsai, C.-S., et al. (2021) · Nature Medicine · doi:10.1038/s41591-021-01506-3 · Cited by 842 · see citing works
Rauniyar, A., Hagos, D. H., Jha, D., Håkegård, J. E., Bagci, U., Rawat, D. B., & Vlassov, V. (2024) · IEEE Internet of Things Journal · doi:10.1109/jiot.2023.3329061 · Cited by 315 · see citing works
“FL is being increasingly explored in medical image analysis to train the DL models on the large data sets distributed across multicenter and cross-border [133], [134], [135].”
BackgroundDo Gradient Inversion Attacks Make Federated Learning Unsafe?
Hatamizadeh, A., Yin, H., Molchanov, P. A., Myronenko, A., Li, W., Dogra, P., Feng, A., Flores, M. G., Kautz, J., Xu, D., et al. (2023) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2023.3239391 · Cited by 133 · see citing works
Secure neuroimaging analysis using federated learning with homomorphic encryption
Stripelis, D., Saleem, H., Ghai, T., Dhinagar, N. J., Gupta, U., Anastasiou, C., Steeg, G. V., Ravi, S., Naveed, M., Thompson, P. M., et al. (2021) · International Symposium on Medical Information Processing and Analysis (SIPAIM) · doi:10.1117/12.2606256 · Cited by 72 · see citing works
“…on distributed and collaborative learning in biomedical imaging have focused on federated gradient averaging and multivariate linear modeling, 4,5 learning from non-IID or unbalanced data at the participating sites, 6 as well as adaptive weighting, asynchronous and semi-synchronous schemes…”
BackgroundA comparative study of federated learning methods for COVID-19 detection
Darzidehkalani, E., Sijtsema, N. M., & van Ooijen, P. M. A. (2024) · Scientific Reports · doi:10.1038/s41598-024-54323-2 · Cited by 31 · see citing works
“These studies have pointed to certain limitations of existing algorithms, such as high communication overhead 17 , as well as convergence issues or catastrophic forgetting when the number of participating hospitals increases 3,18 .”
BackgroundA Federated Learning Approach to Tumor Detection in Colon Histology Images
Gunesli, G. N., Bilal, M., Raza, S. E. A., & Rajpoot, N. (2023) · Journal of Medical Systems · doi:10.1007/s10916-023-01994-5 · Cited by 20 · see citing works
“Likewise, if we focus on the performance results of models trained with different cdr parameters when fdr parameter is equal to 0 (as in FedAvg ) and equal to 0.3 (as the reported model here), we also see that the models trained with fdr = 0.3 give better results than their counterparts ( fdr = 0 ).”
InfluentialWu, J., Dong, Q., Zhang, J., Su, Y., Wu, T., Caselli, R. J., Reiman, E. M., Ye, J., Leporé, N., Chen, K., et al. (2021) · Frontiers in Neuroscience · doi:10.3389/fnins.2021.762458 · Cited by 13 · see citing works
“Some other recent studies focus on multivariate linear modeling (Silva et al., 2020), federated gradient averaging (Remedios et al., 2020), and unbalanced data for multi-site (Yeganeh et al., 2020).”
BackgroundWeighting Schemes for Federated Learning in Heterogeneous and Imbalanced Segmentation Datasets
Otálora, S., Rafael‐Patiño, J., Madrona, A., Fischi-Gómez, E., Ravano, V., Kober, T., Christensen, S., Hakim, A., Wiest, R., Richiardi, J., et al. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-33842-7_4 · Cited by 3 · see citing works
Towards Understanding the Risks of Gradient Inversion in Federated LearningPreprint
Hatamizadeh, A., Yin, H., Molchanov, P. A., Myronenko, A., Li, W., Dogra, P., Feng, A., Flores, M. G., Kautz, J., Xu, D., et al. (2021) · Research Square · doi:10.21203/rs.3.rs-1147182/v1 · Cited by 3 · see citing works
Roth, H. R., Rieke, N., Albarqouni, S., & Li, Q. (2023) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2023.3278528 · Cited by 3 · see citing works
Lin, Y. J., & Komperda, I. (2026) · SN Comprehensive Clinical Medicine · doi:10.1007/s42399-026-02330-x · Cited by 3 · see citing works
FedCKD: KD-based Classifier Refinement for Non-IID Data in Federated Learning
Shen, F., Su, W., Xu, C., Zhang, J., Chen, Y., & He, Y. (2025) · Information Communication Technologies Conference (ICTC) · doi:10.1109/ictc67375.2025.11292468 · Cited by 1 · see citing works
Presentation Attack Detection for Multispectral Face Biometric System Using Federated Learning
Purushothaman, M. M., Adapa, S. R., & Bellamkonda, S. (2024) · Information systems engineering and management · doi:10.1007/978-3-031-69201-7_22 · Cited by 1 · see citing works
Towards Understanding the Risks of Gradient Inversion in Federated LearningPreprint
Hatamizadeh, A., Yin, H., Molchanov, P. A., Myronenko, A., Li, W., Dogra, P., Feng, A., Flores, M. G., Kautz, J., Xu, D., et al. (2021) · Research Square · doi:10.21203/rs.3.rs-1147182/v2 · Cited by 1 · see citing works
Innocenti, L., Antonelli, M., Sebastien Ourselin, A., & Lorenzi, M. (2025) · Elsevier eBooks · doi:10.1016/b978-0-44-323761-4.00034-1 · Cited by 1 · see citing works
Federated Deep Unfolding for Sparse Recovery
Mogilipalepu, K. K., Modukuri, S. K., Madapu, A., & Chepuri, S. P. (2021) · European Signal Processing Conference (EUSIPCO) · doi:10.23919/eusipco54536.2021.9615969
“An important advantage of the FL approach is the decoupling of model training from the essence of direct access to the raw training data [16, 18, 19].”
BackgroundDeep learning using molecular image of chemical structure
Matsuzaka, Y., & Uesawa, Y. (2023) · Elsevier eBooks · doi:10.1016/b978-0-443-18638-7.00005-0
Optimization Analysis of Power Grid Data Quality by Federal Average Algorithm
Tang, L., Li, B., Zeng, T., Liu, H., & Shen, Y. (2025) · International Journal of High Speed Electronics and Systems · doi:10.1142/s0129156425402840
Autism Spectrum Disorder Identification Using Dual-Branch Fusion Model with Privacy-Preserving
Jing, H., Chen, W., Xu, Q., Yang, J., Bandara, D., Wang, R., Zhao, Z., & Liu, C. (2025) · Annals of Data Science · doi:10.1007/s40745-025-00603-1
- Authors
- Remedios, S. W., Butman, J. A., Landman, B. A., & Pham, D. L.
- Affiliations
- National Institutes of Health (US)
- Vanderbilt University (US)
- Henry M. Jackson Foundation (US)
- Corresponding author
- Remedios, Samuel W.
- Venue
- Lecture Notes in Computer Science
- Publisher
- Springer International Publishing
- Published
- 2020
- Volume
- 12444
- Pages
- 170-180
- ISSN
- 0302-9743 (ISSN-L)
- 1611-3349
- PMC
- PMC8442829
- OpenAlex
- W3088370288
- Cited by
- 19
- References
- 33
- Citations by year
- 2026: 1 · 2025: 4 · 2024: 2 · 2023: 6 · 2021: 6
- FWCI
- 1.15
- Type
- conference paper
- Language
- en
- Open access
- green
- europepmc.org
- Licence
- Locations
- Lecture notes in computer science (publishedVersion)
- PubMed (publishedVersion)
- Europe PMC (PubMed Central) (acceptedVersion)
- PubMed Central (acceptedVersion)
- Funders
- National Institute of Biomedical Imaging and Bioengineering (R01EB017230)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Privacy-Preserving Technologies in Data
- Physical Sciences › Computer Science › Artificial Intelligence › Stochastic Gradient Optimization Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Keywords
- machine learning in medicine, data sharing between institutions, handwritten digit recognition
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
