Online crowdsourcing for efficient rating of speech: A validation study
Open accessThe DOI matches its Crossref record; 6 of the 6 references it cites were found in a registry or an index.
Abstract
Blinded listener ratings are essential for valid assessment of interventions for speech disorders, but collecting these ratings can be time-intensive and costly. This study evaluated the validity of speech ratings obtained through online crowdsourcing, a potentially more efficient approach. 100 words from children with /r/ misarticulation were electronically presented for binary rating by 35 phonetically trained listeners and 205 naïve listeners recruited through the Amazon Mechanical Turk (AMT) crowdsourcing platform. Bootstrapping was used to compare different-sized samples of AMT listeners against a “gold standard” (mode across all trained listeners) and an “industry standard” (mode across bootstrapped samples of 3 trained listeners). There was strong overall agreement between trained and AMT listeners. The “industry standard” level of performance was matched by bootstrapped samples with n = 9 AMT listeners. These results support the hypothesis that valid ratings of speech data can be obtained in an efficient manner through AMT. Researchers in communication disorders could benefit from increased awareness of this method.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Evaluating Online Labor Markets for Experimental Research: Amazon.com's Mechanical Turk
Berinsky, A. J., Huber, G. A., & Lenz, G. S. (2012) · Political Analysis · doi:10.1093/pan/mpr057 · Cited by 4,008
Running experiments on Amazon Mechanical Turk
Paolacci, G., Chandler, J., & Ipeirotis, P. G. (2010) · Judgment and Decision Making · doi:10.1017/s1930297500002205 · Cited by 3,767 · cited 1×
“Researchers in behavioral psychology have reported strong agreement between AMT subjects and traditional lab participants in replications of several classic experiments (Crump et al., 2013; Horton, Rand, & Zeckhauser, 2011) or direct comparisons (Paolacci et al., 2010).”
Conducting behavioral research on Amazon’s Mechanical Turk
Mason, W., & Suri, S. (2012) · Behavior Research Methods · doi:10.3758/s13428-011-0124-6 · Cited by 3,172 · cited 5×
“This section summarizes this body of work for the audience of researchers in communication sciences and disorders, drawing in particular on Mason and Suri (2012). The original ‘‘Mechanical Turk’’ was a supposed chess-playing automaton that gained fame in the late 1700s for defeating such formidable opponents as Benjam…”
InfluentialData collection in a flat world: the strengths and weaknesses of Mechanical Turk samples
Goodman, J. K., Cryder, C. E., & Cheema, A. (2013) · Journal of Behavioral Decision Making · doi:10.1002/bdm.1753 · Cited by 2,242 · cited 1×
“In their investigation of the properties of ratings drawn from large numbers of nonexpert raters, Ipeirotis et al. (2014) reported that increasing the number of raters yielded improvements in rating performance only for items within a specific band of rating difficulty.”
Evaluating amazon’s mechanical turk as a tool for experimental behavioral research
Crump, M. J. C., McDonnell, J. V., & Gureckis, T. M. (2013) · PLoS ONE · doi:10.1371/journal.pone.0057410 · Cited by 1,617
Using Mechanical Turk to Study Clinical Populations
Shapiro, D. N., Chandler, J., & Mueller, P. A. (2013) · Clinical Psychological Science · doi:10.1177/2167702612469015 · Cited by 1,139
Cited by
Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems
(2021) · doi:10.1145/3411764 · Cited by 1,240
Conducting Clinical Research Using Crowdsourced Convenience Samples
Chandler, J., & Shapiro, D. (2016) · Annual Review of Clinical Psychology · doi:10.1146/annurev-clinpsy-021815-093623 · Cited by 1,072
Functional segregation of cortical regions underlying speech timing and articulation
Long, M. A., Katlowitz, K. A., Svirsky, M. A., Clary, R. C., Byun, T. M., Majaj, N., Oya, H., Howard, M. A., & Greenlee, J. D. W. (2016) · Neuron · doi:10.1016/j.neuron.2016.01.032 · Cited by 151
“…cooling on speech quality for all recorded sound files, we used an online crowdsourcing approach adapted from a recently validated method (McAllister Byun et al., 2015), in which each vocalization was rated on a visual analog scale (VAS) (Munson et al., 2012) from 0 (extremely degraded) to 1…”
Self-citationAutomatic Assessment of Speech Intelligibility for Individuals with Aphasia
Le, D., Licata, K., Persad, C., & Provost, E. M. (2016) · IEEE/ACM Transactions on Audio, Speech, and Language Processing · doi:10.1109/taslp.2016.2598428 · Cited by 59
An Efficient Deep Learning Based Method for Speech Assessment of Mandarin-Speaking Aphasic Patients
Mahmoud, S. S., Kumar, A., Tang, Y., Li, Y., Gu, X., Fu, J., & Fang, Q. (2020) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2020.3011104 · Cited by 36
“Nevertheless, a previous study has shown that an untrained listener of a language can evaluate the lucidity of speech close to a specialist level [51].”
BackgroundRhythm perception, speaking rate entrainment, and conversational quality: a mediated model
Wynn, C. J., Barrett, T. S., & Borrie, S. A. (2022) · Journal of Speech, Language, and Hearing Research · doi:10.1044/2022_jslhr-21-00293 · Cited by 31
“Participants were recruited using Amazon Mechanical Turk (MTurk; http:// www.mturk.com), an online crowdsourcing website successfully used in social science research involving listener ratings of speech recordings (e.g., Byun et al., 2015; Parker & Borrie, 2018; Yoho et al., 2019).”
MethodPreston, J. L., McAllister, T., Phillips, E., Boyce, S., Tiede, M., Kim, J. S., & Whalen, D. H. (2018) · Journal of Speech, Language, and Hearing Research · doi:10.1044/2018_jslhr-s-17-0441 · Cited by 29
“…the current study collected binary ratings of each speech token from at least nine online listeners, following the protocol introduced in McAllister Byun et al. (2015).3 All participants had U.S.-based IP addresses and, per self-report, were native speakers of English with no history of…”
Computer-Assisted Challenge Point Intervention for Residual Speech Errors
McAllister, T., Hitchcock, E. R., & Ortiz, J. A. (2021) · Perspectives of the ASHA Special Interest Groups · doi:10.1044/2020_persp-20-00191 · Cited by 24
Telepractice Treatment of Residual Rhotic Errors Using App-Based Biofeedback: A Pilot Study
Peterson, L., Savarese, C., Campbell, T., Ma, Z., Simpson, K. O., & McAllister, T. (2022) · Language, Speech, and Hearing Services in Schools · doi:10.1044/2021_lshss-21-00084 · Cited by 16
Reproducible Speech Research With the Artificial Intelligence–Ready PERCEPT Corpora
Benway, N. R., Preston, J. L., Hitchcock, E., Rose, Y., Salekin, A., Liang, W., & McAllister, T. (2023) · Journal of Speech, Language, and Hearing Research · doi:10.1044/2023_jslhr-22-00343 · Cited by 12
Stipancic, K. L., van Brenk, F., Qiu, M., & Tjaden, K. (2025) · Journal of Speech, Language, and Hearing Research · doi:10.1044/2024_jslhr-24-00354 · Cited by 11
Metadata
- Authors
- McAllister Byun, T., Halpin, P. F., & Szeredi, D.
- Venue
- Journal of Communication Disorders
- Publisher
- Elsevier BV
- Published
- 2015-01
- Volume
- 53
- Pages
- 70-83
- ISSN
- 0021-9924 (ISSN-L)
- OpenAlex
- W2059216864
- Cited by
- 95
- References
- 6
- FWCI
- 12.96
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.