Giving speech a hand: Gesture modulates activity in auditory cortex during speech perception
Open accessThe DOI matches its Crossref record; 12 of the 12 references it cites were found in a registry or an index.
Abstract
Viewing hand gestures during face‐to‐face communication affects speech perception and comprehension. Despite the visible role played by gesture in social interactions, relatively little is known about how the brain integrates hand gestures with co‐occurring speech. Here we used functional magnetic resonance imaging (fMRI) and an ecologically valid paradigm to investigate how beat gesture—a fundamental type of hand gesture that marks speech prosody—might impact speech perception at the neural level. Subjects underwent fMRI while listening to spontaneously‐produced speech accompanied by beat gesture, nonsense hand movement, or a still body; as additional control conditions, subjects also viewed beat gesture, nonsense hand movement, or a still body all presented without speech. Validating behavioral evidence that gesture affects speech perception, bilateral nonprimary auditory cortex showed greater activity when speech was accompanied by beat gesture than when speech was presented alone. Further, the left superior temporal gyrus/sulcus showed stronger activity when speech was accompanied by beat gesture than when speech was accompanied by nonsense hand movement. Finally, the right planum temporale was identified as a putative multisensory integration site for beat gesture and speech (i.e., here activity in response to speech accompanied by beat gesture was greater than the summed responses to speech alone and beat gesture alone), indicating that this area may be pivotally involved in synthesizing the rhythmic aspects of both speech and gesture. Taken together, these findings suggest a common neural substrate for processing speech and gesture, likely reflecting their joint communicative role in social interactions. Hum Brain Mapp, 2009. © 2008 Wiley‐Liss, Inc.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Rizzolatti, G., & Craighero, L. (2004) · Annual Review of Neuroscience · doi:10.1146/annurev.neuro.27.070203.144230 · Cited by 7,433 · cited 3×
“Greater activity for this contrast was also observed in visual cortices, associated with sensory processing, as well as in bilateral premotor and left parietal regions, perhaps reflecting ‘‘mirror neuron system’’ activity associated with the perception of meaningful actions [Rizzolatti and Craighero, 2004].”
BackgroundHearing lips and seeing voices
MCGURK, H., & MACDONALD, J. (1976) · Nature · doi:10.1038/264746a0 · Cited by 6,183
Automatic 3D intersubject registration of MR volumetric data in standardized Talairach space
Collins, D. L., Neelin, P., Peters, T. M., & Evans, A. C. (1994) · Journal of Computer Assisted Tomography · doi:10.1097/00004728-199403000-00005 · Cited by 3,431 · cited 1×
“Functional images for each participant were realigned to correct for head motion, normalized into MNI space [Collins et al., 1994; Mazziotta et al., 2001] and smoothed with a 7 mm Gaussian kernel.”
MethodCattaneo, L., & Rizzolatti, G. (2009) · Archives of Neurology · doi:10.1001/archneurol.2009.41 · Cited by 3,137
Visual Contribution to Speech Intelligibility in Noise
Sumby, W. H., & Pollack, I. (1954) · The Journal of the Acoustical Society of America · doi:10.1121/1.1907309 · Cited by 2,583 · cited 1×
“Behavioral effects related to visual speech are similar to those observed for gesture, as concordant visual speech can aid speech perception [Sumby and Pollack, 1954] whereas discordant visual speech can alter auditory perception [McGurk and MacDonald, 1976].”
BackgroundMcNeill, D. (2005) · doi:10.7208/chicago/9780226514642.001.0001 · Cited by 2,157 · cited 2×
“As the benefits of segregating gesture into strict categories has recently come under scrutiny [McNeill, 2005], in order to maintain ecological validity, beat gesture (i.e., rhythmic gesture) was not limited to flicks of the hand for the purposes of this study.”
BackgroundCalvert, G. A., Campbell, R., & Brammer, M. J. (2000) · Current Biology · doi:10.1016/s0960-9822(00)00513-3 · Cited by 954
Cook, S. W., Mitchell, Z., & Goldin-Meadow, S. (2008) · Cognition · doi:10.1016/j.cognition.2007.04.010 · Cited by 500
Song and speech: brain regions involved with perception and covert production
Callan, D. E., Tsytsarev, V., Hanakawa, T., Callan, A. M., Katsuhara, M., Fukuyama, H., & Turner, R. (2006) · NeuroImage · doi:10.1016/j.neuroimage.2006.01.036 · Cited by 287
Self-citationBeyond superior temporal cortex: Intersubject correlations in narrative speech comprehension
Wilson, S. M., Molnar-Szakacs, I., & Iacoboni, M. (2008) · Cerebral Cortex · doi:10.1093/cercor/bhm049 · Cited by 281
Self-citation
Cited by
Price, C. J. (2012) · NeuroImage · doi:10.1016/j.neuroimage.2012.04.062 · Cited by 2,159
The anatomy of language: a review of 100 fMRI studies published in 2009
Price, C. J. (2010) · Annals of the New York Academy of Sciences · doi:10.1111/j.1749-6632.2010.05444.x · Cited by 1,387
When do gestures communicate? A meta-analysis.
Hostetter, A. B. (2011) · Psychological Bulletin · doi:10.1037/a0022128 · Cited by 558
Özyürek, A., Willems, R. M., Kita, S., & Hagoort, P. (2007) · Journal of Cognitive Neuroscience · doi:10.1162/jocn.2007.19.4.605 · Cited by 303
N400 ERPs for actions: building meaning in context
Amoruso, L., Gelormini, C., Aboitiz, F., Alvarez González, M., Manes, F., Cardona, J. F., & Ibanez, A. (2013) · Frontiers in Human Neuroscience · doi:10.3389/fnhum.2013.00057 · Cited by 125
“Consistent with these findings, recent work on language and gesture processing (Willems et al., 2007; Holle et al., 2008; Dick et al., 2009; Hubbard et al., 2009; Kircher et al., 2009) also points to the engagement of temporal areas (STS, STG, MTG), inferior parietal (AG), IFG, and premotor regions…”
ResultGesture Facilitates the Syntactic Analysis of Speech
Holle, H., Obermeier, C., Schmidt-Kassow, M., Friederici, A. D., Ward, J., & Gunter, T. C. (2012) · Frontiers in Psychology · doi:10.3389/fpsyg.2012.00074 · Cited by 70
Prominence in speech and gesture favour second language novel word learning
Kushch, O., Igualada, A., & Prieto, P. (2018) · Language, Cognition and Neuroscience · doi:10.1080/23273798.2018.1435894 · Cited by 45
“Hubbard et al. (2009) found that beat gestures, and not nonsense movements or still images, enhanced auditory processing of speech.”
BackgroundSpeech-accompanying gestures are not processed by the language-processing mechanisms
Jouravlev, O., Zheng, D., Balewski, Z., Le Arnz Pongos, A., Levan, Z., Goldin-Meadow, S., & Fedorenko, E. (2019) · Neuropsychologia · doi:10.1016/j.neuropsychologia.2019.107132 · Cited by 39
Kelly, S. D., Hirata, Y., Manansala, M., & Huang, J. (2014) · Frontiers in Psychology · doi:10.3389/fpsyg.2014.00673 · Cited by 37
Exploring the boundaries of gesture-speech integration during language comprehension
Kelly, S. D. (2017) · Gesture Studies · doi:10.1075/gs.7.12kel · Cited by 30
(2010) · doi:10.1044/1092-4388(2009/08-0243
Metadata
- Authors
- Hubbard, A. L., Wilson, S. M., Callan, D. E., & Dapretto, M.
- Venue
- Human Brain Mapping
- Publisher
- Wiley
- Published
- 2008-04-15
- Volume
- 30
- Issue
- 3
- Pages
- 1028-1037
- ISSN
- 1065-9471 (ISSN-L)
- 1097-0193
- OpenAlex
- W2143584830
- Cited by
- 133
- References
- 12
- FWCI
- 9.56
- Type
- journal article
- Language
- en
- Open access
- bronze
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.