Thousands of Voices for HMM-Based Speech Synthesis–Analysis and Application of TTS Systems Built on Various ASR Corpora
Open accessThe DOI matches its Crossref record; 66 of the 73 references it cites were found in a registry or an index.
Abstract
In conventional speech synthesis, large amounts of phonetically balanced speech data recorded in highly controlled recording studio environments are typically required to build a voice. Although using such data is a straightforward solution for high quality synthesis, the number of voices available will always be limited, because recording costs are high. On the other hand, our recent experiments with HMM-based speech synthesis systems have demonstrated that speaker-adaptive HMM-based speech synthesis (which uses an “average voice model” plus model adaptation) is robust to non-ideal speech data that are recorded under various conditions and with varying microphones, that are not perfectly clean, and/or that lack phonetic balance. This enables us to consider building high-quality voices on “non-TTS” corpora such as ASR corpora. Since ASR corpora generally include a large number of speakers, this leads to the possibility of producing an enormous number of voices automatically. In this paper, we demonstrate the thousands of voices for HMM-based speech synthesis that we have made from several popular ASR corpora such as the Wall Street Journal (WSJ0, WSJ1, and WSJCAM0), Resource Management, Globalphone, and SPEECON databases. We also present the results of associated analysis based on perceptual evaluation, and discuss remaining issues.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“Furthermore, these MCDs are comparable to those observed with text-to-speech synthesizers16 and better than those in previous neural-decoding work with participants that were able to speak naturally11.”
From A high-performance neuroprosthesis for speech decoding and avatar control (2023) · cites this paper as [16]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Maximum Likelihood from Incomplete Data Via the EM Algorithm
Dempster, A. P., Laird, N. M., & Rubin, D. B. (1977) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1977.tb01600.x · Cited by 50,444
Kawahara, H., Masuda-Katsuse, I., & de Cheveigné, A. (1999) · Speech Communication · doi:10.1016/s0167-6393(98)00085-5 · Cited by 1,831
Maximum likelihood linear transformations for HMM-based speech recognition
Gales, M. J. F. (1998) · Computer Speech & Language · doi:10.1006/csla.1998.0043 · Cited by 1,482
Attractive Faces Are Only Average
Langlois, J. H., & Roggman, L. A. (1990) · Psychological Science · doi:10.1111/j.1467-9280.1990.tb00079.x · Cited by 1,317
Pitch-synchronous waveform processing techniques for text-to-speech synthesis using diphones
Moulines, E., & Charpentier, F. (1990) · Speech Communication · doi:10.1016/0167-6393(90)90021-z · Cited by 1,244
Statistical parametric speech synthesis
Zen, H., Tokuda, K., & Black, A. W. (2009) · Speech Communication · doi:10.1016/j.specom.2009.04.004 · Cited by 1,195
Self-citationThe design for the wall street journal-based CSR corpus
Paul, D. B., & Baker, J. M. (1992) · Proceedings of the workshop on Speech and Natural Language - HLT '91 · doi:10.3115/1075527.1075614 · Cited by 1,099
Cox, T., & Cox, M. (2000) · Chapman & Hall/CRC eBooks · doi:10.1201/9780367801700 · Cited by 745
Tree-based state tying for high accuracy acoustic modelling
Young, S. J., Odell, J. J., & Woodland, P. C. (1994) · Proceedings of the workshop on Human Language Technology - HLT '94 · doi:10.3115/1075812.1075885 · Cited by 664
A robust parser for spoken language understanding
Wang, Y.-Y. (1999) · 6th European Conference on Speech Communication and Technology · doi:10.21437/eurospeech.1999-513 · Cited by 653
Distance measures for speech processing
Gray, A., & Markel, J. (1976) · IEEE Transactions on Acoustics, Speech, and Signal Processing · doi:10.1109/tassp.1976.1162849 · Cited by 639
The Application of Hidden Markov Models in Speech Recognition
Gales, M., & Young, S. (2007) · now publishers, Inc. eBooks · doi:10.1561/9781601981219 · Cited by 557
The Application of Hidden Markov Models in Speech Recognition
Gales, M., & Young, S. (2007) · Foundations and Trends® in Signal Processing · doi:10.1561/2000000004 · Cited by 496
A compact model for speaker-adaptive training
Anastasakos, T., McDonough, J., Schwartz, R., & Makhoul, J. (2002) · Proceeding of Fourth International Conference on Spoken Language Processing. ICSLP '96 · doi:10.1109/icslp.1996.607807 · Cited by 476
A Speech Parameter Generation Algorithm Considering Global Variance for HMM-Based Speech Synthesis
TODA, T., & TOKUDA, K. (2007) · IEICE Transactions on Information and Systems · doi:10.1093/ietisy/e90-d.5.816 · Cited by 358
Self-citationWSJCAMO: a British English speech corpus for large vocabulary continuous speech recognition
Robinson, T., Fransen, J., Pye, D., Foote, J., & Renals, S. (2002) · 1995 International Conference on Acoustics, Speech, and Signal Processing · doi:10.1109/icassp.1995.479278 · Cited by 353
The HMM-based speech synthesis system (HTS) version 2.0.
Zen, H., Nose, T., Yamagishi, J., Sako, S., Masuko, T., Black, A. W., & Tokuda, K. (2007) · SSW · Cited by 343
Self-citationAn adaptive algorithm for mel-cepstral analysis of speech
Fukada, T., Tokuda, K., Kobayashi, T., & Imai, S. (1992) · [Proceedings] ICASSP-92: 1992 IEEE International Conference on Acoustics, Speech, and Signal Processing · doi:10.1109/icassp.1992.225953 · Cited by 328
Self-citationYamagishi, J., Kobayashi, T., Nakano, Y., Ogata, K., & Isogai, J. (2009) · IEEE Transactions on Audio, Speech, and Language Processing · doi:10.1109/tasl.2008.2006647 · Cited by 324
Self-citationMDL-based context-dependent subword modeling for speech recognition.
Shinoda, K., & Watanabe, T. (2000) · Journal of the Acoustical Society of Japan (E) · doi:10.1250/ast.21.79 · Cited by 244
Mel-generalized cepstral analysis - a unified approach to speech spectral estimation
Tokuda, K., Kobayashi, T., Masuko, T., & Imai, S. (1994) · 3rd International Conference on Spoken Language Processing (ICSLP 1994) · doi:10.21437/icslp.1994-275 · Cited by 237
Self-citationDetails of the Nitech HMM-Based Speech Synthesis System for the Blizzard Challenge 2005
ZEN, H., TODA, T., NAKAMURA, M., & TOKUDA, K. (2007) · IEICE Transactions on Information and Systems · doi:10.1093/ietisy/e90-1.1.325 · Cited by 222
Self-citationA Hidden Semi-Markov Model-Based Speech Synthesis System
ZEN, H., TOKUDA, K., MASUKO, T., KOBAYASIH, T., & KITAMURA, T. (2007) · IEICE Transactions on Information and Systems · doi:10.1093/ietisy/e90-d.5.825 · Cited by 209
Self-citationGlobalphone: a multilingual speech and text database developed at karlsruhe university
Schultz, T. (2002) · 7th International Conference on Spoken Language Processing (ICSLP 2002) · doi:10.21437/icslp.2002-151 · Cited by 195
Robust Speaker-Adaptive HMM-Based Text-to-Speech Synthesis
Yamagishi, J., Nose, T., Zen, H., Ling, Z.-H., Toda, T., Tokuda, K., King, S., & Renals, S. (2009) · IEEE Transactions on Audio, Speech, and Language Processing · doi:10.1109/tasl.2009.2016394 · Cited by 178
Self-citationKing, S., & Karaiskos, V. (2009) · The Blizzard Challenge 2009 · doi:10.21437/blizzard.2009-1 · Cited by 176
Self-citationAverage-Voice-Based Speech Synthesis Using HSMM-Based Speaker Adaptation and Adaptive Training
YAMAGISHI, J., & KOBAYASHI, T. (2007) · IEICE Transactions on Information and Systems · doi:10.1093/ietisy/e90-d.2.533 · Cited by 170
Self-citationSimultaneous Modeling Of Spectrum, Pitch And Duration In HMM-Based Speech Synthesis
Yoshimura, T., Tokuda, K., Kobayashi, T., Masuko, T., & Kitamura, T. (1999) · Cited by 166
Self-citationMultisyn: Open-domain unit selection for the Festival speech synthesis system
Clark, R. A., Richmond, K., & King, S. (2007) · Speech Communication · doi:10.1016/j.specom.2007.01.014 · Cited by 150
Self-citationThe ATR Multilingual Speech-to-Speech Translation System
Nakamura, S., Markov, K., Nakaiwa, H., Kikui, G., Kawai, H., Jitsuhiro, T., Zhang, J., Yamamoto, H., Sumita, E., & Yamamoto, S. (2006) · IEEE Transactions on Audio Speech and Language Processing · doi:10.1109/tsa.2005.860774 · Cited by 146
SPEECON – Speech Databases for Consumer Devices: Database Specification and Validation
Iskra, D. J., Grosskopf, B., Marasek, K. P., van den Heuvel, H., Diehl, F., & Kiessling, A. (2002) · Language Resources and Evaluation Conference (LREC) · doi:10.63317/3qdvv58t5zk4 · Cited by 135
Speech synthesis using HMMs with dynamic features
Masuko, T., Tokuda, K., Kobayashi, T., & Imai, S. (2002) · 1996 IEEE International Conference on Acoustics, Speech, and Signal Processing Conference Proceedings · doi:10.1109/icassp.1996.541114 · Cited by 129
Self-citation1993 benchmark tests for the ARPA spoken language program
Pallett, D. S., Fiscus, J. G., Fisher, W. M., Garofolo, J. S., Lund, B. A., & Przybocki, M. A. (1994) · DARPA Workshop on Speech and Natural Language (HLT) · doi:10.3115/1075812.1075824 · Cited by 125
XIMERA : A new TTS from ATR based on corpus-based technologies
Kawai, H., Toda, T., Ni, J., Tsuzaki, M., & Tokuda, K. (2004) · Cited by 119
Self-citationTree-based State Tying for High Accuracy Acoustic Modeling
Young, S. J. (1994) · International Conference on Acoustics, Speech, and Signal Processing · Cited by 95
The development of the HTK Broadcast News transcription system: An overview
Woodland, P. C. (2002) · Speech Communication · doi:10.1016/s0167-6393(01)00059-0 · Cited by 93
On the security of HMM-based speaker verification systems against imposture using synthetic speech
Masuko, T., Hitotsumatsu, T., Tokuda, K., & Kobayashi, T. (1999) · 6th European Conference on Speech Communication and Technology (Eurospeech 1999) · doi:10.21437/eurospeech.1999-286 · Cited by 82
Self-citationState mapping based method for cross-lingual speaker adaptation in HMM-based speech synthesis
Wu, Y.-J., Nankaku, Y., & Tokuda, K. (2009) · Interspeech (USB) · doi:10.21437/interspeech.2009-192 · Cited by 76
Self-citationStatistical analysis of the Blizzard Challenge 2007 listening test results
Clark, R. A., Podsiadło, M., Fraser, M., Mayo, C., & King, S. (2007) · Cited by 73
Self-citationThe design of the newspaper-based Japanese large vocabulary continuous speech recognition corpus
Itou, K., Yamamoto, M., Takeda, K., Takezawa, T., Matsuoka, T., Kobayashi, T., Shikano, K., & Itahashi, S. (1998) · International Conference on Spoken Language Processing (ICSLP) · doi:10.21437/icslp.1998-683 · Cited by 64
Imposture using synthetic speech against speaker verification based on spectrum and pitch
Masuko, T., Tokuda, K., & Kobayashi, T. (2000) · 6th International Conference on Spoken Language Processing (ICSLP 2000) · doi:10.21437/icslp.2000-269 · Cited by 61
Self-citationYamagishi, J., Zen, H., Wu, Y.-J., Toda, T., & Tokuda, K. (2008) · The Blizzard Challenge · doi:10.21437/blizzard.2008-7 · Cited by 61
Self-citationText to Speech Synthesis: New Paradigms and Advances
Narayanan, S. S., & Alwan, A. A. (2004) · Cited by 59
An HMM-Based Mandarin Chinese Text-To-Speech System
Qian, Y., Soong, F. K., Chen, Y., & Chu, M. (2006) · Lecture notes in computer science · doi:10.1007/11939993_26 · Cited by 50
The prototype model in speaker identification
Lavner, Y., Rosenhouse, J., & Gath, I. (1999) · European Conference on Speech Communication and Technology (Eurospeech) · doi:10.21437/eurospeech.1999-200 · Cited by 49
The Handbook of Japanese Linguistics
(2001) · Blackwell Publishing Ltd eBooks · doi:10.1111/b.9780631234944.2001.00003.x · Cited by 49
Robustness of HMM-based speech synthesis
Yamagishi, J., Ling, Z.-H., & King, S. (2008) · Interspeech (USB) · doi:10.21437/interspeech.2008-171 · Cited by 44
Self-citationCross-Lingual Speaker Adaptation for HMM-Based Speech Synthesis
Wu, Y.-J., King, S., & Tokuda, K. (2008) · International Symposium on Chinese Spoken Language Processing (ISCSLP) · doi:10.1109/chinsl.2008.ecp.14 · Cited by 42
Self-citationFraser, M., & King, S. (2007) · Cited by 41
Self-citationRevisiting the security of speaker verification systems against imposture using synthetic speech
De Leon, P. L., Apsingekar, V. R., Pucher, M., & Yamagishi, J. (2010) · 2010 IEEE International Conference on Acoustics, Speech and Signal Processing · doi:10.1109/icassp.2010.5495413 · Cited by 37
Self-citationIntra-lingual and cross-lingual voice conversion using harmonic plus stochastic models
Eslava, D. E. (2008) · Dialnet (Universidad de la Rioja) · Cited by 31
DARPA resource management benchmark test results June 1990
Pallett, D. S., Fiscus, J., & Garofolo, J. S. (1990) · DARPA Workshop on Speech and Natural Language (HLT) · doi:10.3115/116580.116683 · Cited by 30
Adaptive training for robust ASR
Gales, M. (2005) · IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU) · doi:10.1109/asru.2001.1034578 · Cited by 28
Correlation analysis of subjective and objective measures for speech quality
Barnwell, T. P., III (2005) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.1980.1171078 · Cited by 28
Measuring the Gap Between HMM-Based ASR and TTS
Dines, J., Yamagishi, J., & King, S. (2010) · IEEE Journal of Selected Topics in Signal Processing · doi:10.1109/jstsp.2010.2079315 · Cited by 25
Self-citationComputer Synthesized Speech Technologies: Tools for Aiding Impairment
Mullennix, J. W., & STERN, S. E. (2010) · doi:10.4018/978-1-61520-725-1 · Cited by 25
Optimizing segment label boundaries for statistical speech synthesis
Black, A. W., & Kominek, J. (2009) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2009.4960451 · Cited by 23
Thousands of voices for HMM-based speech synthesis
Yamagishi, J., Usabaev, B., King, S., Watts, O., Dines, J., Tian, J., Hu, R., Guan, Y., Oura, K., Tokuda, K., et al. (2009) · Interspeech (USB) · doi:10.21437/interspeech.2009-140 · Cited by 20
Self-citationReformulating the HMM as a Trajectory Model
Tokuda, K., Zen, H., & Kitamura, T. (2004) · Cited by 19
Self-citationVasilis Karaiskos, B., King, S., Clark, R. A. J., & Mayo, C. (2008) · The Blizzard Challenge · doi:10.21437/blizzard.2008-1 · Cited by 17
Self-citationAnalysis of speaking styles by two-dimensional visualization of aggregate of acoustic models
Shozakai, M., & Nagino, G. (2004) · Interspeech (USB) · doi:10.21437/interspeech.2004-279 · Cited by 15
QMOS - a Robust Visualization Method for Speaker Dependencies With Different Microphones
Maier, A., Schuster, M. E., Eysholdt, U., Haderlein, T., Cincarek, T., Steidl, S., Batliner, A. M., Wenhardt, S., & Nöth, E. (2009) · Journal of Pattern Recognition Research · doi:10.13176/11.112 · Cited by 10
Tsakalidis, S., & Byrne, B. (2006) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2005.1415150 · Cited by 8
Yamagishi, J., Lincoln, M., King, S., Dines, J., Gibson, M., Tian, J., & Guan, Y. (2009) · The Blizzard Challenge · doi:10.21437/blizzard.2009-7 · Cited by 7
Self-citationBarra-Chicote, R., Montero, J. M., Lutfi, S. L., Macías-Guarasa, J., Yamagishi, J., & King, S. (2008) · Hispana · Cited by 6
Self-citationA Unified and Automatic Approach Of Mandarin HTS System
Guan, Y., Tian, J., Wu, Y.-J., Yamagishi, J., & Nurminen, J. (2010) · ERA · Cited by 2
Self-citation
Research works printed but not linked: 9
- guan, 2009, evaluation of flat start labeling for phoneme based mandarin hts system, Proc ORIENTAL-COCOSDA-09, 187
- 2008, Deliverable Report D2 1
- guan, 2010, an efficient and unified approach of mandarin hts system, Proc ICASSP96
- kawai, 2006, ximera: a concatenative speech synthesis system with large scale corpora, IEICE Trans Inf Syst, j89 d ii, 2688
- tsakalidis, 2005, Cross-corpus normalization of diverse acoustic training data for robust HMM training
- pallet, 1990, darpa resource management bench, Proc Workshop Speech Natural Lang, 298
- tokuda, 0, The HMM-Based Speech Synthesis System (HTS) Version 2 1
- 10.1109/TASL.2009.2016394 · DOI · Paper page
- 10.1109/JSTSP.2010.2079315 · DOI
Cited by
A high-performance neuroprosthesis for speech decoding and avatar control
Metzger, S. L., Littlejohn, K. T., Silva, A. B., Moses, D. A., Seaton, M. P., Wang, R., Dougherty, M. E., Liu, J. R., Wu, P., Berger, M. A., et al. (2023) · Nature · doi:10.1038/s41586-023-06443-4 · Cited by 522
“The final 1D transpose convolution had the same number of kernels as the input signal (16).”
BackgroundVeaux, C., Yamagishi, J., & King, S. (2013) · 2013 International Conference Oriental COCOSDA held jointly with 2013 Conference on Asian Spoken Language Research and Evaluation (O-COCOSDA/CASLRE) · doi:10.1109/icsda.2013.6709856 · Cited by 398
“This technique, referred to as speaker adaptation, has already been successfully applied to “non traditional TTS” corpora such as ASR corpora, which contains a large number of speakers [4].”
MethodSelf-citationSpeech Synthesis Based on Hidden Markov Models
Tokuda, K., Nankaku, Y., Toda, T., Zen, H., Yamagishi, J., & Oura, K. (2013) · Proceedings of the IEEE · doi:10.1109/jproc.2013.2251852 · Cited by 404
“Speaker-adaptive HMMbased synthesis requires as little as 5–7 min of recorded speech from a target speaker to generate a personalized synthetic voice [24]; hence, the average voice model can be easily transformed into a synthetic voice for any number of speakers [25].”
BackgroundSelf-citationEvaluation of Speaker Verification Security and Detection of HMM-Based Synthetic Speech
De Leon, P. L., Pucher, M., Yamagishi, J., Hernaez, I., & Saratxaga, I. (2012) · IEEE Transactions on Audio, Speech, and Language Processing · doi:10.1109/tasl.2012.2201472 · Cited by 231
Self-citationVoice Conversion Using Dynamic Kernel Partial Least Squares Regression
Helander, E., Silen, H., Virtanen, T., & Gabbouj, M. (2012) · IEEE Transactions on Audio, Speech, and Language Processing · doi:10.1109/tasl.2011.2165944 · Cited by 126
“the eigenvoice approach [11] or speaker adaptation in HMM-base d speech synthesis [16].”
Method
Metadata
- Authors
- Yamagishi, J., Usabaev, B., King, S., Watts, O., Dines, J., Tian, J., Guan, Y., Hu, R., Oura, K., Wu, Y.-J., Tokuda, K., Karhila, R., & Kurimo, M.
- Affiliations
- University of Edinburgh (GB)
- University of Tübingen (DE)
- Idiap Research Institute (CH)
- Nokia (China) (CN)
- Nagoya Institute of Technology (JP)
- Microsoft (United States) (US)
- Venue
- IEEE Transactions on Audio, Speech, and Language Processing
- Published
- 2010-07
- Volume
- 18
- Issue
- 5
- Pages
- 984-1004
- ISSN
- 1558-7916 (ISSN-L)
- 1558-7924
- OpenAlex
- W2165143604
- Cited by
- 78
- References
- 73
- Citations by year
- 2026: 1 · 2025: 2 · 2023: 4 · 2022: 3 · 2021: 2 · 2020: 4 · 2019: 1 · 2018: 2 · 2017: 7 · 2016: 7 · 2015: 4 · 2014: 9 · 2013: 6 · 2012: 15
- FWCI
- 10.83
- Type
- journal article
- Language
- en
- Open access
- green
- hdl.handle.net
- Locations
- IEEE Transactions on Audio Speech and Language Processing (publishedVersion)
- ERA (submittedVersion)
- Edinburgh Research Explorer (University of Edinburgh) (submittedVersion)
- Funders
- Helsingin Yliopisto
- European Commission (213845)
- Universidad Politécnica de Madrid
- Engineering and Physical Sciences Research Council (FP7/2007–2013)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
- Physical Sciences › Computer Science › Signal Processing › Speech and Audio Processing
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Phonetics and Phonology Research
- Keywords
- HMM-based speech synthesis, model adaptation, perceptual evaluation, Wall Street Journal, resource management
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.