Online crowdsourcing for efficient rating of speech: A validation study
Open accessThe DOI matches its Crossref record; 6 of the 6 references it cites were found in a registry or an index.
Abstract
Blinded listener ratings are essential for valid assessment of interventions for speech disorders, but collecting these ratings can be time-intensive and costly. This study evaluated the validity of speech ratings obtained through online crowdsourcing, a potentially more efficient approach. 100 words from children with /r/ misarticulation were electronically presented for binary rating by 35 phonetically trained listeners and 205 naïve listeners recruited through the Amazon Mechanical Turk (AMT) crowdsourcing platform. Bootstrapping was used to compare different-sized samples of AMT listeners against a “gold standard” (mode across all trained listeners) and an “industry standard” (mode across bootstrapped samples of 3 trained listeners). There was strong overall agreement between trained and AMT listeners. The “industry standard” level of performance was matched by bootstrapped samples with n = 9 AMT listeners. These results support the hypothesis that valid ratings of speech data can be obtained in an efficient manner through AMT. Researchers in communication disorders could benefit from increased awareness of this method.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Evaluating Online Labor Markets for Experimental Research: Amazon.com's Mechanical Turk
Berinsky, A. J., Huber, G. A., & Lenz, G. S. (2012) · Political Analysis · doi:10.1093/pan/mpr057 · Cited by 4,008
Running experiments on Amazon Mechanical Turk
Paolacci, G., Chandler, J., & Ipeirotis, P. G. (2010) · Judgment and Decision Making · doi:10.1017/s1930297500002205 · Cited by 3,767 · cited 1×
“Researchers in behavioral psychology have reported strong agreement between AMT subjects and traditional lab participants in replications of several classic experiments (Crump et al., 2013; Horton, Rand, & Zeckhauser, 2011) or direct comparisons (Paolacci et al., 2010).”
Conducting behavioral research on Amazon’s Mechanical Turk
Mason, W., & Suri, S. (2012) · Behavior Research Methods · doi:10.3758/s13428-011-0124-6 · Cited by 3,172 · cited 5×
“This section summarizes this body of work for the audience of researchers in communication sciences and disorders, drawing in particular on Mason and Suri (2012). The original ‘‘Mechanical Turk’’ was a supposed chess-playing automaton that gained fame in the late 1700s for defeating such formidable opponents as Benjam…”
InfluentialData collection in a flat world: the strengths and weaknesses of Mechanical Turk samples
Goodman, J. K., Cryder, C. E., & Cheema, A. (2013) · Journal of Behavioral Decision Making · doi:10.1002/bdm.1753 · Cited by 2,242 · cited 1×
“In their investigation of the properties of ratings drawn from large numbers of nonexpert raters, Ipeirotis et al. (2014) reported that increasing the number of raters yielded improvements in rating performance only for items within a specific band of rating difficulty.”
Evaluating amazon’s mechanical turk as a tool for experimental behavioral research
Crump, M. J. C., McDonnell, J. V., & Gureckis, T. M. (2013) · PLoS ONE · doi:10.1371/journal.pone.0057410 · Cited by 1,617
Using Mechanical Turk to Study Clinical Populations
Shapiro, D. N., Chandler, J., & Mueller, P. A. (2013) · Clinical Psychological Science · doi:10.1177/2167702612469015 · Cited by 1,139
Cited by
Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems
(2021) · doi:10.1145/3411764 · Cited by 1,240
Conducting Clinical Research Using Crowdsourced Convenience Samples
Chandler, J., & Shapiro, D. (2016) · Annual Review of Clinical Psychology · doi:10.1146/annurev-clinpsy-021815-093623 · Cited by 1,072
Functional segregation of cortical regions underlying speech timing and articulation
Long, M. A., Katlowitz, K. A., Svirsky, M. A., Clary, R. C., Byun, T. M., Majaj, N., Oya, H., Howard, M. A., & Greenlee, J. D. W. (2016) · Neuron · doi:10.1016/j.neuron.2016.01.032 · Cited by 151
“…cooling on speech quality for all recorded sound files, we used an online crowdsourcing approach adapted from a recently validated method (McAllister Byun et al., 2015), in which each vocalization was rated on a visual analog scale (VAS) (Munson et al., 2012) from 0 (extremely degraded) to 1…”
Self-citationAutomatic Assessment of Speech Intelligibility for Individuals with Aphasia
Le, D., Licata, K., Persad, C., & Provost, E. M. (2016) · IEEE/ACM Transactions on Audio, Speech, and Language Processing · doi:10.1109/taslp.2016.2598428 · Cited by 59
An Efficient Deep Learning Based Method for Speech Assessment of Mandarin-Speaking Aphasic Patients
Mahmoud, S. S., Kumar, A., Tang, Y., Li, Y., Gu, X., Fu, J., & Fang, Q. (2020) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2020.3011104 · Cited by 36
“Nevertheless, a previous study has shown that an untrained listener of a language can evaluate the lucidity of speech close to a specialist level [51].”
Background
Metadata
- Authors
- McAllister Byun, T., Halpin, P. F., & Szeredi, D.
- Venue
- Journal of Communication Disorders
- Publisher
- Elsevier BV
- Published
- 2015-01
- Volume
- 53
- Pages
- 70-83
- ISSN
- 0021-9924 (ISSN-L)
- OpenAlex
- W2059216864
- Cited by
- 95
- References
- 6
- FWCI
- 12.96
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Online crowdsourcing for efficient rating of speech: A validation study. https://app.cochat.ai/papers/10.1016/j.jcomdis.2014.11.003
- The paper
- McAllister Byun, T., Halpin, P. F., & Szeredi, D. (2015). Online crowdsourcing for efficient rating of speech: A validation study. Journal of Communication Disorders. https://doi.org/10.1016/j.jcomdis.2014.11.003
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.