Can Prosody Aid the Automatic Classification of Dialog Acts in Conversational Speech?
Open accessThe DOI matches its Crossref record; 44 of the 44 references it cites were found in a registry or an index.
Abstract
Identifying whether an utterance is a statement, question, greeting, and so forth is integral to effective automatic understanding of natural dialog. Little is known, however, about how such dialog acts (DAs) can be automatically classified in truly natural conversation. This study asks whether current approaches, which use mainly word information, could be improved by adding prosodic information. The study is based on more than 1000 conversations from the Switchboard corpus. DAs were hand-annotated, and prosodic features (duration, pause, F0, energy, and speaking rate) were automatically extracted for each DA. In training, decision trees based on these features were inferred; trees were then applied to unseen test data to evaluate performance. Performance was evaluated for prosody models alone, and after combining the prosody models with word information—either from true words or from the output of an automatic speech recognizer. For an overall classification task, as well as three subtasks, prosody made significant contributions to classification. Feature-specific analyses further revealed that although canonical features (such as F0 for questions) were important, less obvious features could compensate if canonical features were removed. Finally, in each task, integrating the prosodic model with a DA-specific statistical language model improved performance over that of the language model alone, especially for the case of recognized words. Results suggest that DAs are redundantly marked in natural conversation, and that a variety of automatically extractable prosodic features could aid dialog processing in speech applications.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
A Coefficient of Agreement for Nominal Scales
Cohen, J. (1960) · Educational and Psychological Measurement · doi:10.1177/001316446002000104 · Cited by 42,398 · cited 2×
“Interlabeler reliability. Interlabeler reliability on our main (transcript-only) set of annotations was assessed using the Kappa statistic ( Cohen, 1960; Siegel & Castellan, 1988; Carletta, 1996), or the ratio of the proportion of times that raters agree (corrected for chance agreement) to the maximum proportion of ti…”
MethodNonparametric Statistics for the Behavioral Sciences
Guitton, H., & Siegel, S. (1958) · Revue économique · doi:10.2307/3498751 · Cited by 35,245
Classification and Regression Trees.
Gordon, A. D., Breiman, L., Friedman, J. H., Olshen, R. A., & Stone, C. J. (1984) · Biometrics · doi:10.2307/2530946 · Cited by 23,522
Classification and Regression Trees.
Ryzin, J. V., Breiman, L., Friedman, J. H., Olshen, R. A., & Stone, C. J. (1986) · Journal of the American Statistical Association · doi:10.2307/2288003 · Cited by 20,825
Classification and regression trees
Praagman, J. (1985) · European Journal of Operational Research · doi:10.1016/0377-2217(85)90321-2 · Cited by 10,053
A Comprehensive Grammar of the English Language
Huddleston, R., Quirk, R., Greenbaum, S., Leech, G., & Svartvik, J. (1988) · Language · doi:10.2307/415437 · Cited by 7,587
A Comprehensive Grammar of the English Language
Beauvais, P. J., Quirk, R., Greenbaum, S., Leech, G., & Svartvik, J. (1987) · College Composition and Communication · doi:10.2307/357731 · Cited by 5,488
Clark, H. H. (1996) · doi:10.1017/cbo9780511620539 · Cited by 5,123
Speaking: from intention to articulation
(1989) · Choice Reviews Online · doi:10.5860/choice.27-1947 · Cited by 5,003
Speaking: From Intention to Articulation
Levelt, W. J. M. (1989) · doi:10.7551/mitpress/6393.001.0001 · Cited by 4,804
Using language: Communicative acts
Clark, H. H. (1996) · Cited by 2,721
Using language: Index of names
Clark, H. H. (1996) · Cited by 2,720
Hierarchical Mixtures of Experts and the EM Algorithm
Jordan, M. I., & Jacobs, R. A. (1994) · Neural Computation · doi:10.1162/neco.1994.6.2.181 · Cited by 2,673
SWITCHBOARD: telephone speech corpus for research and development
Godfrey, J. J., Holliman, E. C., & McDaniel, J. (1992) · [Proceedings] ICASSP-92: 1992 IEEE International Conference on Acoustics, Speech, and Signal Processing · doi:10.1109/icassp.1992.225858 · Cited by 2,134 · cited 2×
“Elizabeth Shriberg SRI International 333 Ravenswood Ave. Menlo Park, CA 94025 Tel: +1-650-859-3798 FAX: +1-650-859-5984 ABSTRACT Identifying whether an utterance is a statement, question,greeting, and so forth is integral to effective automatic understanding of natural dialog.”
Assessing Agreement on Classification Tasks: The Kappa Statistic
Carletta, J. (1996) · arXiv · doi:10.48550/arxiv.cmp-lg/9602004 · Cited by 2,088 · cited 2×
“…reliability on our main (transcript-only) set of annotations was assessed using the Kappa statistic (Cohen, 1960; Siegel & Castellan, 1988; Carletta, 1996), or the ratio of the proportion of times that raters agree (corrected for chan e agreement) to the maximum proportion of times that…”
De Roeck, A. (1995) · Handbook of Pragmatics · doi:10.1075/hop.m.comm2 · Cited by 1,715
Estimation of probabilities from sparse data for the language model component of a speech recognizer
Katz, S. (1987) · IEEE Transactions on Acoustics, Speech, and Signal Processing · doi:10.1109/tassp.1987.1165125 · Cited by 1,637
Bridle, J. S. (1990) · Neurocomputing · doi:10.1007/978-3-642-76153-9_28 · Cited by 1,237
Language-Independent Prosodic Features
Vaissière, J. (1983) · Springer Series in Language and Communication · doi:10.1007/978-3-642-69103-4_5 · Cited by 388
Automatic labeling of prosodic patterns
Wightman, C. W., & Ostendorf, M. (1994) · IEEE Transactions on Speech and Audio Processing · doi:10.1109/89.326607 · Cited by 275
Some intonational characteristics of discourse structure
Grosz, B., & Hirschberg, J. (1992) · 2nd International Conference on Spoken Language Processing (ICSLP 1992) · doi:10.21437/icslp.1992-103 · Cited by 238
Prosodic features at discourse boundaries of different strength
Swerts, M. (1997) · The Journal of the Acoustical Society of America · doi:10.1121/1.418114 · Cited by 225
A prosodic analysis of discourse segments in direction-giving monologues
Hirschberg, J., & Nakatani, C. H. (1996) · Proceedings of the 34th annual meeting on Association for Computational Linguistics - · doi:10.3115/981863.981901 · Cited by 199
Automatic detection of discourse structure for speech recognition and understanding
Jurafsky, D., Bates, R., Coccaro, N., Martin, R., Meteer, M., Ries, K., Shriberg, E., Stolcke, A., Taylor, P., & Ess-Dykema, C. V. (1997) · 1997 IEEE Workshop on Automatic Speech Recognition and Understanding Proceedings · doi:10.1109/asru.1997.658992 · Cited by 154
Self-citationAutomatic linguistic segmentation of conversational speech
Stolcke, A., & Shriberg, E. E. (2002) · International Conference on Spoken Language Processing (ICSLP) · doi:10.1109/icslp.1996.607773 · Cited by 147
Self-citationA Study on Prosody and Discourse Structure in Cooperative Dialogues
Nakajima, S., & Allen, J. F. (1993) · Phonetica · doi:10.1159/000261940 · Cited by 136
Combining multiple estimators of speaking rate
Morgan, N., & Fosler‐Lussier, E. (2002) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.1998.675368 · Cited by 128
Dialogue act classification using language models
Reithinger, N., & Klesen, M. (1997) · European Conference on Speech Communication and Technology (Eurospeech) · doi:10.21437/eurospeech.1997-589 · Cited by 126
Prosody in Speech Understanding Systems
Kompe, R. (1997) · Lecture notes in computer science · doi:10.1007/3-540-63580-7 · Cited by 115
Lexical, Prosodic, and Syntactic Cues for Dialog Acts
Jurafsky, D. S., Shriberg, E. E., Fox, B. J., & Curl, T. S. (1998) · Cited by 110
Self-citationA prosody only decision-tree model for disfluency detection
Shriberg, E. E., Bates, R. A., & Stolcke, A. (1997) · European Conference on Speech Communication and Technology (Eurospeech) · doi:10.21437/eurospeech.1997-626 · Cited by 98
Self-citationCombining multiple knowledge sources for discourse segmentation
Litman, D. J., & Passonneau, R. J. (1995) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.3115/981658.981673 · Cited by 95
Discourse functions of pitch range in spontaneous and read speech
Ayers, G. M. (1994) · The Knowledge Bank (The Ohio State University) · Cited by 94
Prosody and speech recognition
Waibel, A. (1988) · Cited by 89
Speech recognition using on-line estimation of speaking rate
Morgan, N. H., Fosler, E., & Mirghafori, N. (1997) · European Conference on Speech Communication and Technology (Eurospeech) · doi:10.21437/eurospeech.1997-550 · Cited by 87
Utilizing statistical dialogue act processing in VERBMOBIL
Reithinger, N., & Maier, E. (1995) · Annual Meeting of the Association for Computational Linguistics (ACL) · doi:10.3115/981658.981674 · Cited by 81
A Multi-level Model for Recognition of Intonation Labels
Mari Ostendorf, E., & Ross, K. (1997) · Springer eBooks · doi:10.1007/978-1-4612-2258-3_19 · Cited by 79
Nagata, M., & Morimoto, T. (1994) · Speech Communication · doi:10.1016/0167-6393(94)90071-x · Cited by 70
On the perceptual classification of spontaneous and read speech
Blaauw, E. (1995) · LED eBooks · Cited by 69
Dialog act modelling for conversational speech
Stolcke, A., Shriberg, E. E., Bates, R. A., Coccaro, N., Jurafsky, D. S., Martin, R. W., Meteer, M. W., Ries, K., Taylor, P., & Ess-Dykema, C. V. (1998) · ERA · Cited by 68
Self-citationTowards better language models for spontaneous speech
Suhm, B., & Waibel, A. (1994) · International Conference on Spoken Language Processing (ICSLP) · doi:10.21437/icslp.1994-222 · Cited by 59
Prosody/parse scoring and its application in ATIS
Veilleux, N. M., & Mari Ostendorf, E. (1993) · DARPA Workshop on Speech and Natural Language (HLT) · doi:10.3115/1075671.1075749 · Cited by 26
Relationship between discourse structure and dynamic speech rate
Beinum, F. J. K.-V., & van Donzel, M. E. (2002) · International Conference on Spoken Language Processing (ICSLP) · doi:10.1109/icslp.1996.607960 · Cited by 22
Prosodic Features of Utterances in Task-Oriented Dialogues
Nakajima, S., & Tsukada, H. (1997) · Springer eBooks · doi:10.1007/978-1-4612-2258-3_7 · Cited by 9
Cited by
Dialogue Act Modeling for Automatic Tagging and Recognition of Conversational Speech
Stolcke, A., Ries, K., Coccaro, N., Shriberg, E., Bates, R., Jurafsky, D., Taylor, P., Martin, R., Ess-Dykema, C. V., & Meteer, M. (2000) · Computational Linguistics · doi:10.1162/089120100561737 · Cited by 1,108
Self-citationThe Oxford Handbook of Pragmatics
(2017) · Oxford University Press eBooks · doi:10.1093/oxfordhb/9780199697960.001.0001 · Cited by 509
Acoustic correlates of information structure
Breen, M., Fedorenko, E., Wagner, M., & Gibson, E. (2010) · Language and Cognitive Processes · doi:10.1080/01690965.2010.504378 · Cited by 430
Prosody-based automatic segmentation of speech into sentences and topics
Shriberg, E., Stolcke, A., Hakkani-Tür, D., & Tür, G. (2000) · Speech Communication · doi:10.1016/s0167-6393(00)00028-5 · Cited by 424
Self-citationTurn-taking cues in task-oriented dialogue
Gravano, A., & Hirschberg, J. (2011) · Computer Speech & Language · doi:10.1016/j.csl.2010.10.003 · Cited by 254
Metadata
- Authors
- Shriberg, E., Stolcke, A., Jurafsky, D., Coccaro, N., Meteer, M., Bates, R., Taylor, P., Ries, K., Martin, R., & van Ess-Dykema, C.
- Venue
- Language and Speech
- Publisher
- SAGE Publications
- Published
- 1998-07
- Volume
- 41
- Issue
- 3-4
- Pages
- 443-492
- ISSN
- 0023-8309 (ISSN-L)
- 1756-6053
- OpenAlex
- W2112078429
- Cited by
- 307
- References
- 44
- FWCI
- 11.81
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.