The DOI matches its Crossref record; 46 of the 46 references it cites were found in a registry or an index.
Perceptual Evaluation of Voice Quality: Review, Tutorial, and a Framework for Future Research
Abstract
The reliability of listeners’ ratings of voice quality is a central issue in voice research because of the clinical primacy of such ratings and because they are the standard against which other measures are evaluated. However, an extensive literature review indicates that both intrarater and interrater reliability fluctuate greatly from study to study. Further, our own data indicate that ratings of vocal roughness vary widely across individual clinicians, with a single voice often receiving nearly the full range of possible ratings. No model or theoretical framework currently exists to explain these variations, although such a model might guide development of efficient, valid, and standardized clinical protocols for voice evaluation. We propose a theoretical framework that attributes variability in ratings to several sources (including listeners’ backgrounds and biases, the task used to gather ratings, interactions between listeners and tasks, and random error). This framework may guide development of new clinical voice and speech evaluation protocols, ultimately leading to more reliable perceptual ratings and a better understanding of the perceptual qualities of pathological voices.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Statistical principles in experimental design.
Winer, B. J. (1962) · McGraw-Hill Book Company · doi:10.1037/11774-000 · Cited by 26,755
Intraclass correlations: Uses in assessing rater reliability.
Shrout, P. E., & Fleiss, J. L. (1979) · Psychological Bulletin · doi:10.1037/0033-2909.86.2.420 · Cited by 24,532
Statistics for the social sciences
Hays, W. L. (1973) · Medical Entomology and Zoology · Cited by 2,081
Analysis, synthesis, and perception of voice quality variations among female and male talkers
Klatt, D. H., & Klatt, L. C. (1990) · The Journal of the Acoustical Society of America · doi:10.1121/1.398894 · Cited by 1,691
平野, 実. (1981) · Springer eBooks · Cited by 1,663
Clinical measurement of speech and voice
Baken, R. J. (1987) · Cited by 1,465
Differential diagnostic patterns of dysarthria
Darley, F. L., Aronson, A. E., & Brown, J. R. (1969) · Journal of Speech and Hearing Research · doi:10.1044/jshr.1202.246 · Cited by 1,323
Estimation of the reliability of ratings.
Ebel, R. L. (1951) · Psychometrika · doi:10.1007/bf02288803 · Cited by 976
Interrater reliability and agreement of subjective judgments.
Tinsley, H. E., & Weiss, D. J. (1975) · Journal of Counseling Psychology · doi:10.1037/h0076640 · Cited by 852
Speech and Language: Advances in Basic Research and Practice
Lass, N. J. (1979) · Medical Entomology and Zoology · Cited by 621
Perceptual and Acoustic Correlates of Abnormal Voice Qualities
Hammarberg, B., Fritzell, B., Gaufin, J., Sundberg, J., & Wedin, L. (1980) · Acta Oto-Laryngologica · doi:10.3109/00016488009131746 · Cited by 350
Individual Differences in Voice Quality Perception
Kreiman, J., Gerratt, B. R., Precoda, K., & Berke, G. S. (1992) · Journal of Speech, Language, and Hearing Research · doi:10.1044/jshr.3503.512 · Cited by 281
Self-citationComparing Internal and External Standards in Voice Quality Judgments
Gerratt, B. R., Kreiman, J., Antonanzas-Barroso, N., & Berke, G. S. (1993) · Journal of Speech, Language, and Hearing Research · doi:10.1044/jshr.3601.14 · Cited by 280
Self-citationListener Experience and Perception of Voice Quality
Kreiman, J., Gerratt, B. R., & Precoda, K. (1990) · Journal of Speech, Language, and Hearing Research · doi:10.1044/jshr.3301.103 · Cited by 222
Self-citationStuttering and Speech Naturalness
Martin, R. R., Haroldson, S. K., & Triden, K. A. (1984) · Journal of Speech and Hearing Disorders · doi:10.1044/jshd.4901.53 · Cited by 221
Harmonics-to-Noise Ratio and Psychophysical Measurement of the Degree of Hoarseness
Yumoto, E., Sasaki, Y., & Okamura, H. (1984) · Journal of Speech, Language, and Hearing Research · doi:10.1044/jshr.2701.02 · Cited by 173
Berk, R. A. (1979) · PubMed · Cited by 157
Differential Diagnosis of Hoarseness
Isshiki, N., Okamura, H., Tanabe, M., & Morimoto, M. (2009) · Folia Phoniatrica et Logopaedica · doi:10.1159/000263230 · Cited by 146
Speech Waveform Perturbation Analysis: A Perceptual-Acoustical Comparison of Seven Measures
Askenfelt, A. G., & Hammarberg, B. (1986) · Journal of Speech Language and Hearing Research · doi:10.1044/jshr.2901.50 · Cited by 143
Long-term memory for unfamiliar voices
Papcun, G. J., Kreiman, J. E., & Davis, A. B. (1989) · The Journal of the Acoustical Society of America · doi:10.1121/1.397564 · Cited by 141
Self-citationThe Use of Perceptual Methods by New Clinicians for Assessing Voice Quality
Bassich, C. J., & Ludlow, C. L. (1986) · Journal of Speech and Hearing Disorders · doi:10.1044/jshd.5102.125 · Cited by 129
The multidimensional nature of pathologic vocal quality
Kreiman, J. E., Gerratt, B. R., & Berke, G. S. (1994) · The Journal of the Acoustical Society of America · doi:10.1121/1.410277 · Cited by 114
Self-citationSome Waveform and Spectral Features of Vowel Roughness
Deal, R. E., & Emanuel, F. W. (1978) · Journal of Speech and Hearing Research · doi:10.1044/jshr.2102.250 · Cited by 98
Perception of aperiodicities in synthetically generated voices
Hillenbrand, J. M. (1988) · The Journal of the Acoustical Society of America · doi:10.1121/1.396367 · Cited by 97
EVALUATING INTEROBSERVER RELIABILITY OF INTERVAL DATA1
Hopkins, B. L., & Hermann, J. A. (1977) · Journal of Applied Behavior Analysis · doi:10.1901/jaba.1977.10-121 · Cited by 96
Construct Validity of Direct Magnitude Estimation and Interval Scaling of Speech Intelligibility
Schiavetti, N., Metz, D. E., & Sitler, R. W. (1981) · Journal of Speech Language and Hearing Research · doi:10.1044/jshr.2403.441 · Cited by 88
Samar, V. J., & Metz, D. E. (1988) · Journal of Speech Language and Hearing Research · doi:10.1044/jshr.3103.307 · Cited by 81
Laryngeal Analog Synthesis of Jitter and Shimmer Auditory Parameters of Harshness
Wendahl, R. W. (2009) · Folia Phoniatrica et Logopaedica · doi:10.1159/000263059 · Cited by 80
Sheard, C., Adams, R. D., & Davis, P. J. (1991) · Journal of Speech Language and Hearing Research · doi:10.1044/jshr.3402.285 · Cited by 80
Computer Analysis of Hoarseness
Kojima, H., Gould, W. J., Lambiase, A., & Isshiki, N. (1980) · Acta Oto-Laryngologica · doi:10.3109/00016488009127173 · Cited by 80
Interobserver Reliability and Perceptual Ratings
Kearns, K. P., & Simmons, N. N. (1988) · Journal of Speech Language and Hearing Research · doi:10.1044/jshr.3101.131 · Cited by 74
Acoustic and perceptual analysis of vocal dysfunction
Hammarberg, B., Fritzen, B., Gauffin, J., & Sundberg, J. (1986) · Journal of Phonetics · doi:10.1016/s0095-4470(19)30704-1 · Cited by 72
Stuttering and Speech Naturalness
Ingham, R. J., Gow, M. L., & Costello, J. M. (1985) · Journal of Speech and Hearing Disorders · doi:10.1044/jshd.5002.217 · Cited by 69
Comparing discrimination and recognition of unfamiliar voices
Kreiman, J. E., & Papcun, G. J. (1991) · Speech Communication · doi:10.1016/0167-6393(91)90016-m · Cited by 61
Self-citationSome Parameters of Auditory Roughness
Wendahl, R. W. (2009) · Folia Phoniatrica et Logopaedica · doi:10.1159/000263081 · Cited by 56
An evaluation of residue features as correlates of voice disorders
Prosek, R. A., Montgomery, A. A., Walden, B. E., & Hawkins, D. B. (1987) · Journal of Communication Disorders · doi:10.1016/0021-9924(87)90002-5 · Cited by 51
A new index for evaluation of the turbulent noise in pathological voice
Fukazawa, T., El-Assuooty, A., & Honjo, I. (1988) · The Journal of the Acoustical Society of America · doi:10.1121/1.396012 · Cited by 46
Some Spectral Features of ‘Normal’ and Simulated ‘Rough’ Vowels
Emanuel, F. W., & F.E. Sansone, Jr. (2009) · Folia Phoniatrica et Logopaedica · doi:10.1159/000263295 · Cited by 43
Direct Magnitude Estimation and Equal Appearing Interval Scaling of Vowel Roughness
Toner, M. A., & Emanuel, F. W. (1989) · Journal of Speech Language and Hearing Research · doi:10.1044/jshr.3201.78 · Cited by 40
Relationships of Vowel Characteristics to Listener Ratings of Breathiness
Klich, R. J. (1982) · Journal of Speech Language and Hearing Research · doi:10.1044/jshr.2504.574 · Cited by 39
Spectral Noise Levels and Roughness Severity Ratings for Vowels Produced by Male Children
Arnold, K. S., & Emanuel, F. W. (1979) · Journal of Speech Language and Hearing Research · doi:10.1044/jshr.2203.613 · Cited by 27
Spectral Noise Levels and Roughness Ratings for Vowels Produced by Males and Females
Emanuel, F. W., Lively, M. A., & McCoy, J. F. (2009) · Folia Phoniatrica et Logopaedica · doi:10.1159/000263675 · Cited by 26
Effect of Waveform Changes Upon Roughness Perception
Coleman, R. F. (2009) · Folia Phoniatrica et Logopaedica · doi:10.1159/000263514 · Cited by 23
Sapir, S., Aronson, A., & Thomas, J. E. (1986) · Annals of Otology Rhinology & Laryngology · doi:10.1177/000348948609500205 · Cited by 22
Vocal register effects on vowel spectral noise and roughness: Findings for adult females
Emanuel, F. W., & Scarinzi, A. (1979) · Journal of Communication Disorders · doi:10.1016/0021-9924(79)90047-9 · Cited by 8
Relation between voice profile ratings and aerodynamic and acoustic parameters
Moran, M. J., & Gilbert, H. R. (1984) · Journal of Communication Disorders · doi:10.1016/0021-9924(84)90029-7 · Cited by 7
Cited by
Validation of an instrument to measure voice-related quality of life (V-RQOL)
Hogikyan, N. D., & Sethuraman, G. (1999) · Journal of Voice · doi:10.1016/s0892-1997(99)80010-1 · Cited by 1,143
Consensus Auditory-Perceptual Evaluation of Voice: Development of a Standardized Clinical Protocol
Kempster, G. B., Gerratt, B. R., Verdolini Abbott, K., Barkmeier-Kraemer, J., & Hillman, R. E. (2009) · American Journal of Speech-Language Pathology · doi:10.1044/1058-0360(2008/08-0017 · Cited by 1,008
Self-citationWuyts, F. L., Bodt, M. S. D., Molenberghs, G., Remacle, M., Heylen, L., Millet, B., Lierde, K. V., Raes, J., & Heyning, P. H. V. D. (2000) · Journal of Speech, Language, and Hearing Research · doi:10.1044/jslhr.4303.796 · Cited by 578
Kent, R. D. (1996) · American Journal of Speech-Language Pathology · doi:10.1044/1058-0360.0503.07 · Cited by 432
De Bodt, M. S., Wuyts, F. L., Van de Heyning, P. H., & Croux, C. (1997) · Journal of Voice · doi:10.1016/s0892-1997(97)80026-4 · Cited by 432
Metadata
- Authors
- Kreiman, J., Gerratt, B. R., Kempster, G. B., Erman, A., & Berke, G. S.
- Venue
- Journal of Speech, Language, and Hearing Research
- Published
- 1993-02
- Volume
- 36
- Issue
- 1
- Pages
- 21-40
- ISSN
- 1092-4388 (ISSN-L)
- 1558-9102
- OpenAlex
- W2011585205
- Cited by
- 737
- References
- 46
- FWCI
- 2.91
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.