Federated Learning with Dynamic Transformer for Text to Speech
Abstract
Text to speech (TTS) is a crucial task for user interaction, but TTS model training relies on a sizable set of high-quality original datasets. Due to privacy and security issues, the original datasets are usually unavailable directly. Recently, federated learning proposes a popular distributed machine learning paradigm with an enhanced privacy protection mechanism. It offers a practical and secure framework for data owners to collaborate with others, thus obtaining a better global model trained on the larger dataset. However, due to the high complexity of transformer models, the convergence process becomes slow and unstable in the federated learning setting. Besides, the transformer model trained in federated learning is costly communication and limited computational speed on clients, impeding its popularity. To deal with these challenges, we propose the federated dynamic transformer. On the one hand, the performance is greatly improved comparing with the federated transformer, approaching centralize-trained Transformer-TTS when increasing clients number. On the other hand, it achieves faster and more stable convergence in the training phase and significantly reduces communication time. Experiments on the LJSpeech dataset also strongly prove our method's advantage.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
FedDT-TTS greatly improved transformer models' performance in federated learning, reducing total training time by 40%.
“Their model greatly improved transformer models' performance in the federated learning, reducing total training time by 40%.”
From Applications of Federated Learning; Taxonomy, Challenges, and Research Trends (2022) · cites this paper as [96]
Cites
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946 · cited 3× · see citing works
“[27] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” in 3rd International Conference on Learning Representations, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, Y. Bengio and Y. LeCun, Eds., 2015.”
MethodDelving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification
He, K., Zhang, X., Ren, S., & Sun, J. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.123 · Cited by 19,741 · cited 2× · see citing works
“Moreover, in order to stabilize the convergence, we let the new layer initialize with weights divided by a per-layer normalize constant according to He’s initializer [23].”
MethodA fast learning algorithm for deep belief nets
Hinton, G. E., Osindero, S., & Teh, Y.-W. (2006) · Neural Computation · doi:10.1162/neco.2006.18.7.1527 · Cited by 16,336 · cited 1× · see citing works
“The key insight is, by changing the layer-wise training in the wake-sleep algorithm [22], we grow both the encoder and decoder dynamically through the training process.”
BackgroundCommunication-Efficient Learning of Deep Networks from Decentralized Data
McMahan, H. B., Moore, E., Ramage, D., Hampson, S., & Arcas, B. A. Y. (2016) · arXiv · doi:10.48550/arxiv.1602.05629 · Cited by 14,215 · cited 1× · see citing works
“Following the classical federated learning optimization method FedAvg [28], in each communication round, firstly, the server randomly choose M clients composing St, then send the encrypted current global model’s weights to every client in St.”
MethodIdentity mappings in deep residual networks
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46493-0_38 · Cited by 10,741 · cited 1× · see citing works
“Following the residual block in ResNetV2 [24] and its symmetric structure explanation, the Front-LN method puts the layer normalization inside the residual connection and equips it with an additional final-layer normalization before prediction.”
MethodInfluentialAdvances and Open Problems in Federated Learning
Kairouz, P., & McMahan, H. B. (2021) · Foundations and Trends in Machine Learning · doi:10.1561/2200000083 · Cited by 7,772 · cited 1× · see citing works
“To be specific, the main impediments are as follows: Firstly, all distributed learning methods fatally suffer from the slow and easily unstable convergence problem, which is mainly caused by different clients’ non-IID(including quantity skew) [12,21].”
BackgroundSelf-citationFederated Machine Learning: Concept and Applications
Yang, Q., Liu, Y., Chen, T., & Tong, Y. (2019) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3298981 · Cited by 6,322 · cited 1× · see citing works
“When training models in the federated learning setting, participating clients do not upload their local data to the central server; instead, a central aggregator coordinates the optimization procedure among the clients [15, 16].”
MethodFederated learning: Challenges, methods, and future directions
Li, T., Sahu, A. K., Talwalkar, A., & Smith, V. (2020) · IEEE Signal Processing Magazine · doi:10.1109/msp.2020.2975749 · Cited by 6,210 · cited 1× · see citing works
“When training models in the federated learning setting, participating clients do not upload their local data to the central server; instead, a central aggregator coordinates the optimization procedure among the clients [15, 16].”
MethodOn Layer Normalization in the Transformer ArchitecturePreprint
Xiong, R., Yang, Y., He, D., Zheng, K., Zheng, S., Xing, C., Zhang, H.-S., Lan, Y.-Y., Wang, L.-W., & Liu, T.-Y. (2020) · International Conference on Machine Learning · doi:10.48550/arxiv.2002.04745 · Cited by 1,709 · cited 2× · see citing works
“Similarly, in the structure of the transformer, we put the layer normalization in front of the Multi-Head Attention and Feed Forward Network (FFN) [25].”
MethodSecure, privacy-preserving and federated machine learning in medical imaging
Kaissis, G. A., Makowski, M. R., Rückert, D., & Braren, R. F. (2020) · Nature Machine Intelligence · doi:10.1038/s42256-020-0186-1 · Cited by 1,501 · cited 1× · see citing works
“Federated learning has proved to be a compelling framework in different areas, such as optical character recognition (OCR) [17], natural language processing (NLP) [18,19], Medical Imaging [20] and so on.”
BackgroundNeural Speech Synthesis with Transformer Network
Li, N., Liu, S., Liu, Y., Zhao, S., & Liu, M. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33016706 · Cited by 754 · cited 7× · see citing works
“Model Architecture: The transformer model is following [9].”
BackgroundMethodInfluentialICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)
(2021) · IEEE International Conference on Acoustics, Speech, and Signal Processing · doi:10.1109/icassp39728.2021 · Cited by 254 · see citing works
Federated Learning for Vision-and-Language Grounding Problems
Liu, F., Wu, X., Ge, S., Fan, W., & Zou, Y. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i07.6824 · Cited by 115 · cited 1× · see citing works
“Federated learning has proved to be a compelling framework in different areas, such as optical character recognition (OCR) [17], natural language processing (NLP) [18,19], Medical Imaging [20] and so on.”
BackgroundMultiSpeech: Multi-Speaker Text to Speech with Transformer
Chen, M., Tan, X., Ren, Y., Xu, J., Sun, H., Zhao, S., & Qin, T. (2020) · Interspeech 2020 · doi:10.21437/interspeech.2020-3139 · Cited by 92 · see citing works
Aligntts: Efficient Feed-Forward Text-to-Speech System Without Explicit Alignment
Zeng, Z., Wang, J., Cheng, N., Xia, T., & Xiao, J. Z. (2020) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp40776.2020.9054119 · Cited by 65 · cited 1× · see citing works
“Recently, many variants of transformer models such as FastSpeech [8], Transformer-TTS [9], AlignTTS [1], MultiSpeech [10] have shown great promises in the TTS task.”
MethodSelf-citationEmpirical Studies of Institutional Federated Learning For Natural Language Processing
Zhu, X., Wang, J., Hong, Z., & Xiao, J. Z. (2020) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2020.findings-emnlp.55 · Cited by 54 · cited 1× · see citing works
“Federated learning has proved to be a compelling framework in different areas, such as optical character recognition (OCR) [17], natural language processing (NLP) [18,19], Medical Imaging [20] and so on.”
BackgroundSelf-citationTraining Keyword Spotting Models on Non-IID Data with Federated Learning
Hard, A., Partridge, K., Nguyen, C., Subrahmanya, N., Shah, A., Zhu, P., Moreno, I. L., & Mathews, R. (2020) · Interspeech (USB) · doi:10.21437/interspeech.2020-3023 · Cited by 53 · cited 1× · see citing works
“To be specific, the main impediments are as follows: Firstly, all distributed learning methods fatally suffer from the slow and easily unstable convergence problem, which is mainly caused by different clients’ non-IID(including quantity skew) [12,21].”
BackgroundEfficient Client Contribution Evaluation for Horizontal Federated Learning
Zhao, J., Zhu, X., Wang, J., & Xiao, J. (2021) · ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp39728.2021.9413377 · Cited by 50 · see citing works
Self-citationFederated Learning of Unsegmented Chinese Text Recognition Model
Zhu, X., Wang, J., Hong, Z., Xia, T., & Xiao, J. Z. (2019) · IEEE International Conference on Tools with Artificial Intelligence (ICTAI) · doi:10.1109/ictai.2019.00186 · Cited by 42 · see citing works
Self-citationImproving the Prosody of RNN-Based English Text-To-Speech Synthesis by Incorporating a BERT Model
Kenter, T., Sharma, M. K., & Clark, R. (2020) · Interspeech (USB) · doi:10.21437/interspeech.2020-1430 · Cited by 27 · see citing works
A Quantitative Metric for Privacy Leakage in Federated Learning
Liu, Y., Zhu, X., Wang, J.-Z., et al. (2021) · IEEE International Conference on Acoustics, Speech, and Signal Processing · doi:10.1109/icassp39728.2021.9413539 · Cited by 26 · see citing works
Self-citationGraphTTS: Graph-to-Sequence Modelling in Neural Text-to-Speech
Sun, A., Wang, J., Cheng, N., Peng, H., Zeng, Z., & Xiao, J. Z. (2020) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp40776.2020.9053355 · Cited by 16 · see citing works
Self-citationLearning Speaker Embedding from Text-to-Speech
Cho, J., Żelasko, P., Villalba, J., et al. (2020) · Interspeech · doi:10.21437/interspeech.2020-2970 · Cited by 13
Interactive Text-to-Speech System via Joint Style Analysis
Gao, Y., Zheng, W., Yang, Z., Koehler, T., Fuegen, C., & He, Q. (2020) · Interspeech (USB) · doi:10.21437/interspeech.2020-3069 · Cited by 12 · see citing works
GraphPB: Graphical Representations of Prosody Boundary in Speech Synthesis
Sun, A., Wang, J., Cheng, N., Peng, H., Zeng, Z., Kong, L., & Xiao, J. Z. (2021) · IEEE Spoken Language Technology Workshop (SLT) · doi:10.1109/slt48900.2021.9383530 · Cited by 8 · see citing works
Self-citationEnd-to-End Text-to-Speech Synthesis with Unaligned Multiple Language Units Based on Attention
Aso, M., Takamichi, S., & Saruwatari, H. (2020) · Interspeech (USB) · doi:10.21437/interspeech.2020-2347 · Cited by 2 · see citing works
Flagged first, then by citations
3 of the 29 references couldn’t be matched to a known paper.
Cited by
Sorted by citations
Applications of Federated Learning; Taxonomy, Challenges, and Research Trends
Shaheen, M., Farooq, M. S., Umer, T., & Kim, B.-S. (2022) · Electronics · doi:10.3390/electronics11040670 · Cited by 157 · see citing works
“[96] 2021 NLP FL LJSpeech dataset Dynamic transformer (FedDT-TTS) is proposed, where encoder and decoder increase layers dynamically to provide faster convergence with lesser communication cost in FL for the TTS task, and then compare FedT-TTS and FedDT-TTS on an unbalanced dataset.”
MethodEnd-to-End Speech Recognition from Federated Acoustic Models
Gao, Y., Parcollet, T., Zaiem, S., Fernández-Marqués, J., de Gusmão, P. P. B., Beutel, D. J., & Lane, N. D. (2022) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp43922.2022.9747161 · Cited by 41 · see citing works
Transformers: A Security Perspective
Latibari, B. S., Nazari, N., Chowdhury, M. A., Gubbi, K. I., Fang, C., Ghimire, S., Hosseini, E., Sayadi, H., Homayoun, H., Salehi, S., et al. (2024) · IEEE Access · doi:10.1109/access.2024.3509372 · Cited by 41 · see citing works
Adaptive Few-Shot Learning Algorithm for Rare Sound Event Detection
Zhao, C., Wang, J., Li, L., et al. (2022) · 2022 International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn55064.2022.9892604 · Cited by 12 · see citing works
“It doesn’t exhibit any stationary patterns like phoneme in speech [3], [4] or rhythm in music.”
BackgroundSelf-citationZhao, C., Wang, J., Qu, X., et al. (2022) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp43922.2022.9746051 · Cited by 9 · see citing works
Self-citationAdaptive Sparse and Monotonic Attention for Transformer-based Automatic Speech Recognition
Zhao, C., Wang, J., Wei, W., et al. (2022) · IEEE International Conference on Data Science and Advanced Analytics (DSAA) · doi:10.1109/dsaa54385.2022.10032360 · Cited by 6 · see citing works
“recurrent neural networks, Transformer can perform parallel training very well as it abandons the recurrent structure that has an inherently sequential nature [10]–[13].”
BackgroundSelf-citationQSpeech: Low-Qubit Quantum Speech Application Toolkit
Hong, Z., Wang, J., Qu, X., Zhao, C., Wei, T., & Xiao, J. (2022) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn55064.2022.9892496 · Cited by 5 · see citing works
“Additionally, we propose a framework called QSpeech, which uses the QNN module for easy use in speech applications [16], [17].”
MethodSelf-citationDT-SV: A Transformer-based Time-domain Approach for Speaker Verification
Zhang, N., Wang, J., Hong, Z., Zhao, C., Qu, X., & Xiao, J. Z. (2022) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn55064.2022.9892002 · Cited by 6 · see citing works
“Nowadays, Transformer with self-attention mechanism have become an effective model in a variety of application fields [10]–[12], which mainly focus on the processing of deep neural networks on certain areas of feature maps or certain temporal slots, including the scenario of SV [2].”
BackgroundSelf-citationCodeFed: Federated Speech Recognition for Low-Resource Code-Switching Detection
Madan, C., Diddee, H., Kumar, D., & Mittal, M. (2022) · ACM Transactions on Asian and Low-Resource Language Information Processing · doi:10.1145/3571732 · Cited by 1 · see citing works
“…series data, as seen in large-vocabulary speech recognition [37] and the slow and unstable convergence of relatively large speech-pretrained models [38] pose challenges across the wide-spread use of federated training of speech models. he given task calls for the development of a federated…”
BackgroundLearning Invariant Representation and Risk Minimized for Unsupervised Accent Domain Adaptation
Zhao, C., Wang, J., Qu, X., Wang, H., & Xiao, J. Z. (2023) · IEEE Spoken Language Technology Workshop (SLT) · doi:10.1109/slt54892.2023.10022892 · Cited by 1 · see citing works
“Meanwhile, it has been observed that discrete latents obtained in an unsupervised learning framework are high-level speech descriptors that correspond to phonemes [11, 12, 13, 14].”
BackgroundSelf-citationPersonalized Federated FlowChain for Human Trajectory Prediction
Menon, K. I. A., Prehofer, C., Xu, Y., & Singh, S. (2025) · International Conference on Federated Learning Technologies and Applications (FLTA) · doi:10.1109/flta67013.2025.11336406
- Authors
- Hong, Z., Wang, J., Qu, X., Liu, J., Zhao, C., & Xiao, J.
- Venue
- Interspeech 2021
- Publisher
- ISCA
- Published
- 2021-08-30
- Pages
- 3590-3594
- ISSN
- 2308-457X (ISSN-L)
- OpenAlex
- W3196787717
- Cited by
- 11
- References
- 29
- Citations by year
- 2025: 1 · 2024: 1 · 2023: 1 · 2022: 8
- FWCI
- 1.16
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- Interspeech (USB) (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Privacy-Preserving Technologies in Data
- Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
- Physical Sciences › Computer Science › Signal Processing › Speech and Audio Processing
- Keywords
- federated learning, text-to-speech, privacy-preserving machine learning, convergence rate, communication efficiency, client-side computation, LJSpeech
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
