Fast, Compact, and High Quality LSTM-RNN Based Statistical Parametric Speech Synthesizers for Mobile Devices
Open accessThe DOI matches its Crossref record; 3 of the 3 references it cites were found in a registry or an index.
Abstract
Acoustic models based on long short-term memory recurrent neural networks (LSTM-RNNs) were applied to statistical parametric speech synthesis (SPSS) and showed significant improvements in naturalness and latency over those based on hidden Markov models (HMMs). This paper describes further optimizations of LSTM-RNN-based SPSS for deployment on mobile devices; weight quantization, multi-frame inference, and robust inference using an ε-contaminated Gaussian loss function. Experimental results in subjective listening tests show that these optimizations can make LSTM-RNN-based SPSS comparable to HMM-based SPSS in runtime speed while maintaining naturalness. Evaluations between LSTM-RNN- based SPSS and HMM-driven unit selection speech synthesis are also presented.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318
Backpropagation through time: what it does and how to do it
Werbos, P. J. (1990) · Proceedings of the IEEE · doi:10.1109/5.58337 · Cited by 4,879 · cited 1×
“between the target and predicted output features. A distributed CPU implementation of mini-batch asynchronous stochastic gradient descent (ASGD)- based truncated back propagation through time (BPTT) [41] algorithm was used [40]. The learning rates for the acoustic and duration LSTM-RNNs were 10 5 and 10 6, respectivel…”
MethodLong short-term memory recurrent neural network architectures for large scale acoustic modeling
Sak, H., Senior, A., & Beaufays, F. (2014) · Interspeech 2014 · doi:10.21437/interspeech.2014-80 · Cited by 2,973
Cited by
WaveNet: a generative model for raw audio
Oord, A. V. D., Dieleman, S., Zen, H., Simonyan, K., Vinyals, O., Graves, A., Kalchbrenner, N., Senior, A., & Kavukcuoglu, K. (2016) · arXiv · doi:10.48550/arxiv.1609.03499 · Cited by 4,911
“Zen et al. (2016) showed that state-of-the-art statistical parametric speech syntheziers matched state-of-the-art concatenative ones in some languages.”
BackgroundMethodSelf-citationNatural TTS Synthesis by Conditioning Wavenet on MEL Spectrogram Predictions
Shen, J., Pang, R., Weiss, R. J., Schuster, M., Jaitly, N., Yang, Z., Chen, Z., Zhang, Y., Wang, Y., Skerrv-Ryan, R., et al. (2018) · 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2018.8461368 · Cited by 2,791
Deep Learning in Mobile and Wireless Networking: A Survey
Zhang, C., Patras, P., & Haddadi, H. (2019) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2019.2904897 · Cited by 1,768
Tacotron: towards end-to-end speech synthesis
Wang, Y., Skerry-Ryan, R. J., Stanton, D., Wu, Y., Weiss, R. J., Jaitly, N., Yang, Z., Xiao, Y., Chen, Z., Bengio, S., et al. (2017) · Interspeech 2017 · doi:10.21437/interspeech.2017-1452 · Cited by 1,746
“We compare our model with a parametric (based on LSTM (Zen et al., 2016)) and a concatenative system (Gonzalvo et al., 2016), both of which are in production.”
MethodInfluentialSelf-citationASVspoof 2019: A large-scale public database of synthesized, converted and replayed speech
Wang, X., Yamagishi, J., Todisco, M., Delgado, H., Nautsch, A., Evans, N., Sahidullah, M., Vestman, V., Kinnunen, T., Lee, K. A., et al. (2020) · Computer Speech & Language · doi:10.1016/j.csl.2020.101114 · Cited by 505
Self-citationStyle Tokens: Unsupervised Style Modeling, Control and Transfer in End-to-End Speech Synthesis
Wang, Y., Stanton, D., Zhang, Y., Skerry-Ryan, R., Battenberg, E., Shor, J., Xiao, Y., Ren, F., Ye, J., & Saurous, R. A. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1803.09017 · Cited by 497
(2017) · International Conference on Learning Representations · Cited by 314
A survey on neural speech synthesis
Tan, X., Qin, T., Soong, F., & Liu, T.-Y. (2021) · arXiv · doi:10.48550/arxiv.2106.15561 · Cited by 216
“Speedup with Domain Knowledge Domain knowledge from speech can be leveraged to speed up inference, such as linear prediction [357], multiband modeling [411, 400], subscale prediction [147], multi-frame prediction [420, 376, 367, 123], streaming synthesis [74], etc.”
BackgroundSynthetic speech detection through short-term and long-term prediction traces
Borrelli, C., Bestagini, P., Antonacci, F., Sarti, A., & Tubaro, S. (2021) · EURASIP Journal on Information Security · doi:10.1186/s13635-021-00116-3 · Cited by 80
On-device machine learning An algorithms and learning theory perspective
Dhar, S., Guo, J., Liu, J., Tripathi, S., Kurup, U., & Shah, M. (2019) · arXiv · doi:10.48550/arxiv.1911.00623 · Cited by 47
Deepfake Speech Detection: A Spectrogram Analysis
Firc, A., Malinka, K., & Hanáček, P. (2024) · Proceedings of the 39th ACM/SIGAPP Symposium on Applied Computing · doi:10.1145/3605098.3635911 · Cited by 16
“TTS and VC technologies are used, and some synthesis technology includes Variational Autoencoders [14, 43], Gaussian mixture models [25] or LSTM-RNN models [24, 42].”
Method
Metadata
- Authors
- Zen, H., Agiomyrgiannakis, Y., Egberts, N., Henderson, F., & Szczepaniak, P.
- Venue
- Interspeech 2016
- Publisher
- ISCA
- Published
- 2016-09-08
- Pages
- 2273-2277
- ISSN
- 2308-457X (ISSN-L)
- OpenAlex
- W2428702538
- Cited by
- 102
- References
- 3
- FWCI
- 8.89
- Type
- conference paper
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.