Speech Synthesis Evaluation — State-of-the-Art Assessment and Suggestion for a Novel Research Program
Open accessThe DOI matches its Crossref record; 40 of the 40 references it cites were found in a registry or an index.
Abstract
Wagner P, Beskow J, Betz S, et al. Speech Synthesis Evaluation—State-of-the-Art Assessment and Suggestion for a Novel Research Program. In: Proceedings of the 10th Speech Synthesis Workshop (SSW10). 2019.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“It is preferred to correlation-based metrics, such as computing a Pearson correlation between ground truth and decoded acoustic waveforms, as MCD weighs the features in the produced sound that are most perceptually relevant for conveying information151.”
From The speech neuroprosthesis (2024) · cites this paper as [151]
Citations
Cites
WaveNet: a generative model for raw audio
Oord, A. V. D., Dieleman, S., Zen, H., Simonyan, K., Vinyals, O., Graves, A., Kalchbrenner, N., Senior, A., & Kavukcuoglu, K. (2016) · arXiv · doi:10.48550/arxiv.1609.03499 · Cited by 4,911 · cited 3×
“How can novel machine learning and high quality synthesis such as WaveNet be put to use in TTS evaluation?”
MethodBartneck, C., Kulić, D., Croft, E., & Zoghbi, S. (2009) · International Journal of Social Robotics · doi:10.1007/s12369-008-0001-3 · Cited by 3,193 · cited 1×
“To overcome this problem, typical surveys employed in HCI or HRI tend to be very extensive [35], thereby trying to address all potential quality dimensions a user may have employed in her or his assessment.”
BackgroundEncyclopedia of Survey Research Methods
Lavrakas, P. (2008) · doi:10.4135/9781412963947 · Cited by 2,689
Explaining Phonetic Variation: A Sketch of the H&H Theory
Lindblom, B. (1990) · Speech Production and Speech Modelling · doi:10.1007/978-94-009-2037-8_16 · Cited by 1,824
The Meaning of Intonational Contours in the Interpretation of Discourse
Pierrehumbert, J., & Hirschberg, J. (1990) · Intentions in Communication · doi:10.7551/mitpress/3839.003.0016 · Cited by 1,433
The Meaning of Intonational Contours in the Interpretation of Discourse
Pierrehumbert, J. B., & Hirschberg, J. (1990) · doi:10.7916/d8kd24fp · Cited by 830 · cited 1×
“…(tutoring, chatting, arguing, counseling...), individual (hearing problems, attitude, level of distraction, motivation, familiarity), linguistic (frequency, predictability, suprisal, importance) or environmental settings (external noise, mutual visibility, ...) [11, 12, 13, 14, 15, 16, 17, 18].”
Aylett, M., & Turk, A. (2004) · Language and Speech · doi:10.1177/00238309040470010201 · Cited by 781 · cited 1×
“…(tutoring, chatting, arguing, counseling...), individual (hearing problems, attitude, level of distraction, motivation, familiarity), linguistic (frequency, predictability, suprisal, importance) or environmental settings (external noise, mutual visibility, ...) [11, 12, 13, 14, 15, 16, 17, 18].”
Probabilistic relations between words: Evidence from reduction in lexical production
Jurafsky, D., Bell, A., Gregory, M., & Raymond, W. D. (2001) · Typological Studies in Language · doi:10.1075/tsl.45.13jur · Cited by 762
A Bayesian explanation of the ‘Uncanny Valley’ effect and related psychological phenomena
Moore, R. K. (2012) · Scientific Reports · doi:10.1038/srep00864 · Cited by 212 · cited 1×
“This is in line with the analysis by [20, 21], claiming that long-known problems of human machine interaction such as the uncanny valley can be modeled as a mismatch between a user’s expectations and a machine’s actual expression.”
ResultQuality-Net: An End-to-End Non-intrusive Speech Quality Assessment Model Based on BLSTM
Fu, S.-W., Tsao, Y., Hwang, H.-T., & Wang, H.-M. (2018) · Interspeech 2018 · doi:10.21437/interspeech.2018-1802 · Cited by 178
Tic Tac TOE: Effects of predictability and importance on acoustic prominence in language production
Watson, D. G., Arnold, J. E., & Tanenhaus, M. K. (2007) · Cognition · doi:10.1016/j.cognition.2007.06.009 · Cited by 155 · cited 1×
“…(tutoring, chatting, arguing, counseling...), individual (hearing problems, attitude, level of distraction, motivation, familiarity), linguistic (frequency, predictability, suprisal, importance) or environmental settings (external noise, mutual visibility, ...) [11, 12, 13, 14, 15, 16, 17, 18].”
Comprehension of Synthetic Speech Produced by Rule: A Review and Theoretical Interpretation
Duffy, S. A., & Pisoni, D. B. (1992) · Language and Speech · doi:10.1177/002383099203500401 · Cited by 133 · cited 1×
“While some researchers postulate to assess it in content repetition tasks [44], [45] suggests it can only be assessed indirectly, e.”
BackgroundIn defense of stylistic diversity in speech research
Wagner, P., Trouvain, J., & Zimmerer, F. (2014) · Journal of Phonetics · doi:10.1016/j.wocn.2014.11.001 · Cited by 122 · cited 1×
“, speaking loudly may be an optimal choice in a loud pub, but entirely inappropriate in more formal situations [18].”
BackgroundSelf-citationAnalyzing Speech Quality Perception Using Electroencephalography
Voigt‐Antons, J., Schleicher, R., Arndt, S., Möller, S., Porbadnigk, A. K., & Curio, G. (2012) · IEEE Journal of Selected Topics in Signal Processing · doi:10.1109/jstsp.2012.2191936 · Cited by 85 · cited 2×
“Alternatively, metrics target more fine-grained system diagnostics such as multiple stimuli with hidden reference and anchor (MUSHRA) [38], or pairwise comparison approaches that ultimately allow for a multidimensional scaling of systems, but rely on multiple assessments of comparable utterances across systems [39].”
BackgroundAutoMOS: Learning a non-intrusive assessor of naturalness-of-speechPreprint
Patton, B., Agiomyrgiannakis, Y., Terry, M. A., Wilson, K., Saurous, R. A., & Sculley, D. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.09207 · Cited by 78 · cited 1×
“More impressive results were reported by AutoMOS [33], but this system has only been trained and evaluated on a single speaker, and is not publicly available.”
BackgroundQuality of Experience Versus User Experience
Wechsung, I., & Moor, K. D. (2014) · T-labs series in telecommunication services · doi:10.1007/978-3-319-02681-7_3 · Cited by 65 · cited 1×
“Our proposition parallels similar suggestions within the HCI community, striving to enhance the technology-centered concept of “Quality of Experience” with the more user-centered concept of “User Experience” [50].”
ResultInteractive Hesitation Synthesis: Modelling and Evaluation
Betz, S., Carlmeyer, B., Wagner, P., & Wrede, B. (2018) · Multimodal Technologies and Interaction · doi:10.3390/mti2010009 · Cited by 58 · cited 4×
“An embedding in a realistic application can also make interlocutors more sensitive for quality issues: in [5], it was found that an interactive setting increased listeners’ sensitivity to quality losses introduced by synthetic hesitations, even though the same hesitations increased their performance in a memory task.”
BackgroundInfluentialSelf-citationWester, M., Valentini-Botinhao, C., & Henter, G. E. (2015) · Interspeech (USB) · doi:10.21437/interspeech.2015-689 · Cited by 44 · cited 2×
“In a similar vein, recent times have seen an increasing number of papers criticizing traditional approaches to TTS evaluation [7, 5], or pointing out frequent methodological flaws such as the low validity of most TTS evaluations due to small participant numbers and a lack of diversity in the tested listener groups, es…”
BackgroundBias and Statistical Significance in Evaluating Speech Synthesis with Mean Opinion Scores
Rosenberg, A., & Ramabhadran, B. (2017) · Interspeech (USB) · doi:10.21437/interspeech.2017-479 · Cited by 44 · cited 2×
“In a similar vein, recent times have seen an increasing number of papers criticizing traditional approaches to TTS evaluation [7, 5], or pointing out frequent methodological flaws such as the low validity of most TTS evaluations due to small participant numbers and a lack of diversity in the tested listener groups, es…”
BackgroundUsing Pupillometry to Measure the Cognitive Load of Synthetic Speech
Govender, A., & King, S. (2018) · Interspeech (USB) · doi:10.21437/interspeech.2018-1174 · Cited by 43 · cited 1×
“Also, some first attempts of using physiological rather than behavioral metrics such as pupil dilation or EEG exist [49, 39].”
BackgroundRodero, E. (2017) · Computers in Human Behavior · doi:10.1016/j.chb.2017.08.044 · Cited by 42 · cited 1×
“Contrary to this, however, are the results by [24], finding human voices to be preferred in more complex tasks.”
ResultResearch on Diagnostic Evaluation of Speech Intelligibility
Voiers, W. D., Sharpley, A. D., & Hehmsoth, C. J. (1973) · doi:10.21236/ad0755918 · Cited by 41 · cited 1×
“If a researcher is interested in less impressionistic measures of intelligibility, established measures are “semantically unpredictable sentences” (SUSs, [41]), together with word edit distance, word error rate estimates, or rhyme tests [42, 43].”
BackgroundDimensions of Segmental Variability: Interaction of Prosody and Surprisal in Six Languages
Malisz, Z., Brandt, E., Möbius, B., Oh, Y. M., & Andreeva, B. (2018) · Frontiers in Communication · doi:10.3389/fcomm.2018.00025 · Cited by 28 · cited 1×
“…(tutoring, chatting, arguing, counseling...), individual (hearing problems, attitude, level of distraction, motivation, familiarity), linguistic (frequency, predictability, suprisal, importance) or environmental settings (external noise, mutual visibility, ...) [11, 12, 13, 14, 15, 16, 17, 18].”
Self-citationProviding Computer Game Characters with Conversational Abilities
Gustafson, J., Boye, J., Fredriksson, M., Johanneson, L., & Königsmann, J. (2005) · Lecture notes in computer science · doi:10.1007/11550617_4 · Cited by 22 · cited 1×
“Yet another form of behavioral analysis was chosen in [46], where participants’ level of verbal adaptation to different interactive character displays was analyzed in a dialogue task.”
BackgroundSelf-citationBeyond the Listening Test: An Interactive Approach to TTS Evaluation
Mendelson, J., & Aylett, M. P. (2017) · Interspeech 2017 · doi:10.21437/interspeech.2017-1438 · Cited by 21 · cited 1×
“In a similar vein, recent times have seen an increasing number of papers criticizing traditional approaches to TTS evaluation [7, 5], or pointing out frequent methodological flaws such as the low validity of most TTS evaluations due to small participant numbers and a lack of diversity in the tested listener groups, es…”
BackgroundRating Naturalness in Speech Synthesis: The Effect of Style and Expectation
Dall, R., Yamagishi, J., & King, S. (2014) · Speech prosody · doi:10.21437/speechprosody.2014-192 · Cited by 19
Perception of Paralinguistic Traits in Synthesized Voices
Baird, A., Jørgensen, S. H., Parada‐Cabaleiro, E., Hantke, S., Cummins, N., & Schuller, B. W. (2017) · Audio Mostly Conference (AM) · doi:10.1145/3123514.3123528 · Cited by 18 · cited 1×
“In a similar vein, recent times have seen an increasing number of papers criticizing traditional approaches to TTS evaluation [7, 5], or pointing out frequent methodological flaws such as the low validity of most TTS evaluations due to small participant numbers and a lack of diversity in the tested listener groups, es…”
BackgroundModern speech synthesis for phonetic sciences: a discussion and an evaluationPreprint
Malisz, Z., Henter, G. E., Valentini-Botinhao, C., Watts, O., Beskow, J., & Gustafson, J. (2020) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/dxvhc · Cited by 17 · cited 2×
“However, with the advent of high-quality, probabilistic waveform-level synthesis models such as WaveNet [34], we finally have synthesizers capable of generating high-quality speech waveforms [25].”
BackgroundMethodSelf-citationDualKeepon: a human–robot interaction testbed to study linguistic features of speech
Cao, H.-L., Jensen, L. C., Nghiem, X. N., Vu, H. T. T., Beir, A. D., Esteban, P. G., de Perre, G. V., Lefeber, D., & Vanderborght, B. (2018) · Intelligent Service Robotics · doi:10.1007/s11370-018-0266-9 · Cited by 9 · cited 1×
“, preference tests, have already been successfully applied to the evaluation of prosodic styles [26].”
MethodHinterleitner, F. (2017) · T-labs series in telecommunication services · doi:10.1007/978-981-10-3734-4 · Cited by 8
Moers, D., Wagner, P., & Breuer, S. (2007) · SSW · Cited by 7
Self-citationSpeech Synthesis Evaluation – Realizing a Social Turn
Wagner, P., & Betz, S. (2017) · PUB – Publications at Bielefeld University (Bielefeld University) · Cited by 7
Self-citationSynthesized lengthening of function words - The fuzzy boundary between fluency and disfluency
Betz, S., Zarrieß, S., & Wagner, P. (2017) · PUB – Publications at Bielefeld University (Bielefeld University) · Cited by 6
Self-citationAudience response system-based assessment for analysis-by-synthesis
Edlund, J., Tånnander, C., & Gustafson, J. (2015) · Cited by 5
Self-citationHönemann, A., & Wagner, P. (2015) · PUB – Publications at Bielefeld University (Bielefeld University) · Cited by 4
Self-citationHow should Pepper sound - Preliminary investigations on robot vocalizations
Burkhardt, F., Saponja, M., Seßner, J., & Weiß, B. (2019) · Cited by 3
Anmerkungen zu den Begriffen „Verständlichkeit“ und „Verstehbarkeit“ bei der Sprachqualitätsmessung
Fellbaum, K., & Höpfner, D. (2014) · Cited by 2
A diagnostic evaluation of the speech input and output devices in GPS navigation systems
Aschenberner, B., & Wagner, P. (2005) · PUB – Publications at Bielefeld University (Bielefeld University) · Cited by 1
Self-citationEvaluating prosody in synthetic speech with online (eye-tracking) and offline (rating) methods.
Rajkumar, R. P., White, M., Speer, S. R., & Ito, K. (2010) · Cited by 1
Nishimura, R., NAGAO, T., ICHIMANDA, A., & Kitaoka, N. (2018) · Journal of Japan Society for Fuzzy Theory and Intelligent Informatics · doi:10.3156/jsoft.30.6_840 · Cited by 1 · cited 1×
“A more recent study directly used the intelligibility profiles of elderly listeners to fine-tune a TTS to their particular needs [29].”
Method
Cited by
Saka, A. B., Oyedele, L. O., Akanbi, L. A., Ganiyu, S. A., Chan, D. W. M., & Bello, S. A. (2023) · Advanced Engineering Informatics · doi:10.1016/j.aei.2022.101869 · Cited by 161
Conversational AI: Dialogue Systems, Conversational Agents, and Chatbots
McTear, M. (2021) · Synthesis Lectures on Human Language Technologies · doi:10.1007/978-3-031-02176-3 · Cited by 155
Kühne, K., Fischer, M. H., & Zhou, Y. (2020) · Frontiers in Neurorobotics · doi:10.3389/fnbot.2020.593732 · Cited by 163
“While most synthesized voices have reached a high level of intelligibility, it is the perceived quality that still requires clarification (Polkosky and Lewis, 2003; Wagner et al., 2019).”
BackgroundSilva, A. B., Littlejohn, K. T., Liu, J. R., Moses, D. A., & Chang, E. F. (2024) · Nature Reviews Neuroscience · doi:10.1038/s41583-024-00819-9 · Cited by 103
“The performance of TTS systems is often described using mel-cepstral distortion (MCD) and open-ended human-transcribed WER 149,150 .”
Choice of Voices: A Large-Scale Evaluation of Text-to-Speech Voice Quality for Long-Form Content
Cambre, J., Colnago, J., Maddock, J., Tsai, J., & Kaye, J. (2020) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376789 · Cited by 102
“However, speech technology is advancing so rapidly that the state-of-the-art is likely outpacing our own publishing cycles [41,49].”
BackgroundInfluential
Metadata
- Authors
- Wagner, P., Beskow, J., Betz, S., Edlund, J., Gustafson, J., Eje Henter, G., Le Maguer, S., Malisz, Z., Székely, É., Tånnander, C., & Voße, J.
- Affiliations
- Bielefeld University (DE)
- KTH Royal Institute of Technology (SE)
- Corresponding author
- Wagner, Petra
- ORCID
- Venue
- 10th ISCA Workshop on Speech Synthesis (SSW 10)
- Publisher
- ISCA
- Published
- 2019-09-20
- Pages
- 105-110
- OpenAlex
- W3011535310
- Cited by
- 74
- References
- 40
- Citations by year
- 2026: 1 · 2025: 9 · 2024: 12 · 2023: 17 · 2022: 12 · 2021: 11 · 2020: 12
- FWCI
- 3.35
- Type
- conference paper
- Language
- en
- Open access
- green
- nbn-resolving.org
- Licence
- cc-by-nc-nd
- Locations
- ISCA Speech Synthesis Workshop (SSW) (publishedVersion)
- PUB – Publications at Bielefeld University (Bielefeld University) (submittedVersion, cc-by-nc-nd)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
- Physical Sciences › Computer Science › Signal Processing › Speech and Audio Processing
- Health Sciences › Health Professions › Speech and Hearing › Voice and Speech Disorders
- Keywords
- speech synthesis evaluation, speech quality assessment, state-of-the-art assessment, research program proposal, speech synthesis
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.