Beyond the Listening Test: An Interactive Approach to TTS Evaluation
The DOI matches its Crossref record; 2 of the 2 references it cites were found in a registry or an index.
Abstract
Traditionally, subjective text-To-speech (TTS) evaluation is performed through audio-only listening tests, where participants evaluate unrelated, context-free utterances. The ecological validity of ...
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Bartneck, C., Kulić, D., Croft, E., & Zoghbi, S. (2009) · International Journal of Social Robotics · doi:10.1007/s12369-008-0001-3 · Cited by 3,193 · cited 1×
“• The Godspeed Questionnaire, reduced to Anthropomorphism, Animacy, and Likeability, and Intelligence [28].”
BackgroundSocial cues in multimedia learning: role of speaker’s voice
Mayer, R. E., Sobko, K., & Mautone, P. D. (2003) · Journal of Educational Psychology · doi:10.1037/0022-0663.95.2.419 · Cited by 429
Cited by
Voice in Human–Agent Interaction: A Survey
Seaborn, K., Miyake, N. P., Pennefather, P., & Otake-Matsuura, M. (2022) · ACM Computing Surveys · doi:10.1145/3386867 · Cited by 249
“An exception is the work of Mendelson and Aylett [116], which showed a contradiction between voice-only and avatar conditions.”
BackgroundThe State of Speech in HCI: Trends, Themes and Challenges
Clark, L., Doyle, P., Garaialde, D., Gilmartin, E., Schlögl, S., Edlund, J., Aylett, M., Cabral, J., Munteanu, C., Edwards, J., et al. (2019) · Interacting with Computers · doi:10.1093/iwc/iwz016 · Cited by 193
“However, there are a number of studies published in other fields that are relevant to speech HCI work, including cognitive psychology, ergonomics, linguistics, and speech technology (e.g. Branigan et al., 2011; Large et al., 2017; Cowan & Branigan, 2015; Mendelson & Aylett, 2017).”
BackgroundSelf-citationWagner, P., Beskow, J., Betz, S., Edlund, J., Gustafson, J., Eje Henter, G., Le Maguer, S., Malisz, Z., Székely, É., Tånnander, C., et al. (2019) · 10th ISCA Workshop on Speech Synthesis (SSW 10) · doi:10.21437/ssw.2019-19 · Cited by 74
“In a similar vein, recent times have seen an increasing number of papers criticizing traditional approaches to TTS evaluation [7, 5], or pointing out frequent methodological flaws such as the low validity of most TTS evaluations due to small participant numbers and a lack of diversity in the tested listener groups, es…”
BackgroundPersuasive Synthetic Speech: Voice Perception and User Behaviour
Dubiel, M., Halvey, M., Gallegos, P. O., & King, S. (2020) · Proceedings of the 2nd Conference on Conversational User Interfaces · doi:10.1145/3405755.3406120 · Cited by 28
“Despite such listening tests being the standard way to evaluate TTS, a fair criticism is the lack of ecological validity, notably that listeners make their judgements without any context [25, 29].”
Background
Metadata
- Authors
- Mendelson, J., & Aylett, M. P.
- Venue
- Interspeech 2017
- Publisher
- ISCA
- Published
- 2017-08-20
- Pages
- 249-253
- ISSN
- 2308-457X (ISSN-L)
- OpenAlex
- W2748820881
- Cited by
- 21
- References
- 2
- FWCI
- 1.65
- Type
- conference paper
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.