Visual Prosody and Speech Intelligibility: Head Movement Improves Auditory Speech Perception
Abstract
People naturally move their heads when they speak, and our study shows that this rhythmic head motion conveys linguistic information. Three-dimensional head and face motion and the acoustics of a talker producing Japanese sentences were recorded and analyzed. The head movement correlated strongly with the pitch (fundamental frequency) and amplitude of the talker's voice. In a perception study, Japanese subjects viewed realistic talking-head animations based on these movement recordings in a speech-in-noise task. The animations allowed the head motion to be manipulated without changing other characteristics of the visual or acoustic speech. Subjects correctly identified more syllables when natural head motion was present in the animation than when it was eliminated or distorted. These results suggest that nonverbal gestures such as head movements play a more direct role in the perception of speech than previously known.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
People naturally rotate their heads to improve audio perception.
“As posed by Munhall et al. [37], people naturally rotate their heads to improve audio perception.”
From An investigation into the effectiveness of using acoustic touch to assist people who are blind (2023) · cites this paper as [37]
Cites
Visual Contribution to Speech Intelligibility in Noise
Sumby, W. H., & Pollack, I. (1954) · The Journal of the Acoustical Society of America · doi:10.1121/1.1907309 · Cited by 2,583 · see citing works
Bimodal speech perception: an examination across languages
Massaro, D. W., Cohen, M. M., Gesi, A., Heredia, R., & Tsuzaki, M. (1993) · Journal of Phonetics · doi:10.1016/s0095-4470(19)30230-x · Cited by 108 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Interactional synchrony: signals, mechanisms and benefits
Hoehl, S., Fairhurst, M., & Schirmer, A. (2021) · Social Cognitive and Affective Neuroscience · doi:10.1093/scan/nsaa024 · Cited by 231 · see citing works
The Socio-Temporal Brain: Connecting People in Time
Schirmer, A., Meck, W. H., & Penney, T. B. (2016) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2016.08.002 · Cited by 83 · see citing works
Movement Coordination during Conversation
Latif, N., Barbosa, A. V., Vatiokiotis-Bateson, E., Castelhano, M. S., & Munhall, K. G. (2014) · PLoS ONE · doi:10.1371/journal.pone.0105036 · Cited by 72 · see citing works
Self-citationPredicting Speaker Head Nods and the Effects of Affective Information
Lee, J., & Marsella, S. C. (2010) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2010.2051874 · Cited by 37 · see citing works
Modeling multimodal behaviors from speech prosody,
Ding, Y., Pelachaud, C., & Artières, T. (2013) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-40415-3_19 · Cited by 32 · see citing works
An investigation into the effectiveness of using acoustic touch to assist people who are blind
Zhu, H. Y., Hossain, S. N., Jin, C., Singh, A. K., Nguyen, M. T. D., Deverell, L., Nguyen, V., Gates, F. S., Fernandez, I. G., Melencio, M. V., et al. (2023) · PLoS ONE · doi:10.1371/journal.pone.0290431 · Cited by 16 · see citing works
- Authors
- Munhall, K. G., Jones, J. A., Callan, D. E., Kuratate, T., & Vatikiotis-Bateson, E.
- Affiliations
- Queen's University (CA)
- Advanced Telecommunications Research Institute International (JP)
- ATR-International Human Information Science Laboratories, Kyoto, Japan
- Corresponding author
- Munhall, K.G.
- Venue
- Psychological Science
- Publisher
- SAGE Publications
- Published
- 2004-02
- Volume
- 15
- Issue
- 2
- Pages
- 133-137
- ISSN
- 0956-7976 (ISSN-L)
- 1467-9280
- OpenAlex
- W2118004237
- Cited by
- 482
- References
- 2
- Citations by year
- 2026: 16 · 2025: 9 · 2024: 19 · 2023: 18 · 2022: 23 · 2021: 19 · 2020: 16 · 2019: 33 · 2018: 35 · 2017: 29 · 2016: 24 · 2015: 29 · 2014: 24 · 2013: 32 · 2012: 27
- FWCI
- 20.40
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Psychological Science (publishedVersion)
- PubMed (publishedVersion)
- Funders
- National Institute on Deafness and Other Communication Disorders (DC 05774, R01DC005774)
- Topics
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Phonetics and Phonology Research
- Physical Sciences › Computer Science › Signal Processing › Speech and Audio Processing
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Multisensory perception and integration
- Keywords
- visual prosody, head movement, speech intelligibility, speech perception, nonverbal gestures, fundamental frequency, pitch, talking head animation, speech perception in noise, Japanese speech
- MeSH terms
- Adult, Biomechanical Phenomena, Facial Expression, Female, Gestures, Humans, Male, Perceptual Distortion, Phonetics, Semantics, Sound Localization, Sound Spectrography, Speech Acoustics, Speech Intelligibility, Speech Perception, User-Computer Interface, Head Movements, Imaging, Three-Dimensional
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
