Co‐speech gestures influence neural activity in brain regions associated with processing semantic information
Open accessThe DOI matches its Crossref record; 28 of the 28 references it cites were found in a registry or an index.
Abstract
Everyday communication is accompanied by visual information from several sources, including co‐speech gestures, which provide semantic information listeners use to help disambiguate the speaker's message. Using fMRI, we examined how gestures influence neural activity in brain regions associated with processing semantic information. The BOLD response was recorded while participants listened to stories under three audiovisual conditions and one auditory‐only (speech alone) condition. In the first audiovisual condition, the storyteller produced gestures that naturally accompany speech. In the second, the storyteller made semantically unrelated hand movements. In the third, the storyteller kept her hands still. In addition to inferior parietal and posterior superior and middle temporal regions, bilateral posterior superior temporal sulcus and left anterior inferior frontal gyrus responded more strongly to speech when it was further accompanied by gesture, regardless of the semantic relation to speech. However, the right inferior frontal gyrus was sensitive to the semantic import of the hand movements, demonstrating more activity when hand movements were semantically unrelated to the accompanying speech. These findings show that perceiving hand movements during speech modulates the distributed pattern of neural activation involved in both biological motion perception and discourse comprehension, suggesting listeners attempt to find meaning, not only in the words speakers produce, but also in the hand movements that accompany speech. Hum Brain Mapp, 2009. © 2009 Wiley‐Liss, Inc.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Desikan, R. S., Ségonne, F., Fischl, B., Quinn, B. T., Dickerson, B. C., Blacker, D., Buckner, R. L., Dale, A. M., Maguire, R. P., Hyman, B. T., et al. (2006) · NeuroImage · doi:10.1016/j.neuroimage.2006.01.021 · Cited by 13,324 · cited 3×
“3 and 4) were defined on each individual surface representation using an automated parcellation procedure [Desikan et al., 2006; Fischl et al., 2004].”
Cortical Surface-Based Analysis
Dale, A. M., Fischl, B., & Sereno, M. I. (1999) · NeuroImage · doi:10.1006/nimg.1998.0395 · Cited by 10,910
Fischl, B., Salat, D. H., Busa, E., Albert, M., Dieterich, M., Haselgrove, C., van der Kouwe, A., Killiany, R., Kennedy, D., Klaveness, S., et al. (2002) · Neuron · doi:10.1016/s0896-6273(02)00569-x · Cited by 8,610
Rizzolatti, G., & Craighero, L. (2004) · Annual Review of Neuroscience · doi:10.1146/annurev.neuro.27.070203.144230 · Cited by 7,433
Nonparametric permutation tests for functional neuroimaging: A primer with examples
Nichols, T. E., & Holmes, A. P. (2002) · Human Brain Mapping · doi:10.1002/hbm.1058 · Cited by 6,256
Cortical Surface-Based Analysis
Fischl, B., Sereno, M. I., & Dale, A. M. (1999) · NeuroImage · doi:10.1006/nimg.1998.0396 · Cited by 6,147 · cited 4×
“Participants gave written informed consent and the Institutional Review Board of the Biological Sciences Division of The University of Chicago approved the study.”
InfluentialThe Repertoire of Nonverbal Behavior: Categories, Origins, Usage, and Coding
EKMAN, P., & FRIESEN, W. V. (1969) · Semiotica · doi:10.1515/semi.1969.1.1.49 · Cited by 3,316 · cited 2×
“The remaining gestures were categorized as beat or deictic gestures, and none were categorized as codified emblems [Ekman and Friesen, 1969].”
Neurophysiological mechanisms underlying the understanding and imitation of action
Rizzolatti, G., Fogassi, L., & Gallese, V. (2001) · Nature Reviews Neuroscience · doi:10.1038/35090060 · Cited by 3,302
Cattaneo, L., & Rizzolatti, G. (2009) · Archives of Neurology · doi:10.1001/archneurol.2009.41 · Cited by 3,137 · cited 1×
“A number of studies have identified brain regions, including ventral premotor cortex, IFG, and IPL, whose functional properties resemble those of macaque ‘‘mirror neurons,’’ that fire during both execution of one’s own actions and the observation of the same actions of others [for reviews see Iacoboni, 2005; Rizzolatt…”
McNeill, D. (2005) · doi:10.7208/chicago/9780226514642.001.0001 · Cited by 2,157 · cited 7×
“Listeners not only process the words that speakers produce, but also continuously integrate gestures with speech and with other visual information (e.g., the speaker’s lips, mouth, and eyes) to arrive at the speaker’s meaning [GoldinMeadow, 2006; Kendon, 1994; McNeill, 2005].”
InfluentialMeta-analyzing left hemisphere language areas: phonology, semantics, and sentence processing.
Vigneau, M., Beaucousin, V., Hervé, P. Y., Duffau, H., Crivello, F., Houdé, O., Mazoyer, B., & Tzourio-Mazoyer, N. (2006) · NeuroImage · doi:10.1016/j.neuroimage.2005.11.002 · Cited by 1,851
Poldrack, R. A., Wagner, A. D., Prull, M. W., Desmond, J. E., Glover, G. H., & Gabrieli, J. D. E. (1999) · NeuroImage · doi:10.1006/nimg.1999.0441 · Cited by 1,448
Integration of word meaning and world knowledge in language comprehension
Hagoort, P., Hald, L., Bastiaansen, M., & Petersson, K. M. (2004) · Science · doi:10.1126/science.1095455 · Cited by 1,396 · cited 2×
“, 1997], while others point to IFG, particularly on the left [Hagoort et al., 2004; also see Table I in Van Petten and Luka, 2006].”
Face-specific processing in the human fusiform gyrus
McCarthy, G., Puce, A., Gore, J. C., & Allison, T. (1997) · Journal of Cognitive Neuroscience · doi:10.1162/jocn.1997.9.5.605 · Cited by 1,127 · cited 2×
“early visual cortices (%BA 17, 18, 19), in bilateral posterior fusiform gyrus typically active when participants are viewing faces [Kanwisher et al., 1997; McCarthy et al., 1997], and in posterior cortical regions associated with processing biological motion and in crossmodal integration of the auditory and visual mod…”
Crossmodal Processing in the Human Brain: Insights from Functional Neuroimaging Studies
Calvert, G. A. (2001) · Cerebral Cortex · doi:10.1093/cercor/11.12.1110 · Cited by 1,012
Calvert, G. A., Campbell, R., & Brammer, M. J. (2000) · Current Biology · doi:10.1016/s0960-9822(00)00513-3 · Cited by 954
Badre, D., Poldrack, R. A., Paré-Blagoev, E. J., Insler, R. Z., & Wagner, A. D. (2005) · Neuron · doi:10.1016/j.neuron.2005.07.023 · Cited by 914 · cited 2×
“…left hemisphere function for language.2 Several studies of semantic selection in which left IFG was the focus also report activation in right IFG [Badre et al., 2005; Fletcher et al., 2000; Thompson-Schill et al., 1997], and right IFG is consistently implicated in the comprehension of figurative…”
Bilateral brain processes for comprehending natural language
Jung-Beeman, M. (2005) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2005.09.009 · Cited by 908
Listening to speech activates motor areas involved in speech production
Wilson, S. M., Saygin, A. P., Sereno, M. I., & Iacoboni, M. (2004) · Nature Neuroscience · doi:10.1038/nn1263 · Cited by 885 · cited 2×
“First, relative to baseline, bilateral premotor cortex was active in all conditions, which is comparable to that reported in other investigations of audiovisual speech perception and gesture [Calvert and Campbell, 2003; Holle et al., 2008; Pulvermüller et al., 2006; Skipper et al., 2007b; Wilson and Iacoboni, 2006; Wi…”
The extended language network: A meta-analysis of neuroimaging studies on text comprehension
Ferstl, E. C., Neumann, J., Bogler, C., & von Cramon, D. Y. (2008) · Human Brain Mapping · doi:10.1002/hbm.20422 · Cited by 703 · cited 5×
“Semantic information in language is known to be processed diffusely by regions in both the left and right hemispheres [Bookheimer, 2002; Ferstl et al., 2008; Jung-Beeman, 2005], but the contribution of both hemispheres to processing gesture has not been extensively investigated.”
InfluentialElectrophysiology and brain imaging of biological motion
Puce, A., & Perrett, D. (2003) · Philosophical Transactions of the Royal Society of London. Series B: Biological Sciences · doi:10.1098/rstb.2002.1221 · Cited by 693
Neural mechanisms of imitation
Iacoboni, M. (2005) · Current Opinion in Neurobiology · doi:10.1016/j.conb.2005.10.010 · Cited by 606 · cited 3×
“…premotor cortex, IFG, and IPL, whose functional properties resemble those of macaque ‘‘mirror neurons,’’ that fire during both execution of one’s own actions and the observation of the same actions of others [for reviews see Iacoboni, 2005; Rizzolatti and Craighero, 2004; Rizzolatti et al., 2001].”
Validating cluster size inference: random field and permutation methods
Hayasaka, S. (2003) · NeuroImage · doi:10.1016/j.neuroimage.2003.08.003 · Cited by 491
Beyond superior temporal cortex: Intersubject correlations in narrative speech comprehension
Wilson, S. M., Molnar-Szakacs, I., & Iacoboni, M. (2008) · Cerebral Cortex · doi:10.1093/cercor/bhm049 · Cited by 281
Selecting Among Competing Alternatives: Selection and Retrieval in the Left Inferior Frontal Gyrus
Moss, H. E., Abdallah, S., Fletcher, P., Bright, P., Pilgrim, L., Acres, K., & Tyler, L. K. (2005) · Cerebral Cortex · doi:10.1093/cercor/bhi049 · Cited by 244 · cited 2×
“, 2001] or selection of competing semantic meanings [Fletcher et al., 2000; Moss et al., 2005; Thompson-Schill et al., 1997], right IFG has only rarely been implicated specifically in semantic processes.”
Meaningful gestures: Electrophysiological indices of iconic gesture comprehension
Wu, Y. C., & Coulson, S. (2005) · Psychophysiology · doi:10.1111/j.1469-8986.2005.00356.x · Cited by 187 · cited 2×
“…well-characterized N400 deflection, thought to reflect semantic integration [Kutas and Federmeier, 2000; Kutas and Hillyard, 1980], is sensitive to the relationship between gesture and speech [Holle and Gunter, 2007; Kelly et al., 2004, 2007; Özyürek et al., 2007; Wu and Coulson, 2005, 2007a,b].”
How iconic gestures enhance communication: An ERP study
Wu, Y. C., & Coulson, S. (2007) · Brain and Language · doi:10.1016/j.bandl.2006.12.003 · Cited by 150
Brain networks subserving the extraction of sentence information and its encoding to memory
Hasson, U., Nusbaum, H. C., & Small, S. L. (2007) · Cerebral Cortex · doi:10.1093/cercor/bhm016 · Cited by 75 · cited 1×
“These deactivated clusters are consistent with those found in other discourse and language studies [Hasson et al., 2007; Wilson et al., 2008].”
Self-citation
Cited by
Price, C. J. (2012) · NeuroImage · doi:10.1016/j.neuroimage.2012.04.062 · Cited by 2,159
An integrated theory of language production and comprehension
Pickering, M. J., & Garrod, S. (2013) · Behavioral and Brain Sciences · doi:10.1017/s0140525x12001495 · Cited by 1,569
The anatomy of language: a review of 100 fMRI studies published in 2009
Price, C. J. (2010) · Annals of the New York Academy of Sciences · doi:10.1111/j.1749-6632.2010.05444.x · Cited by 1,387
The Neural Bases of Social Cognition and Story Comprehension
Mar, R. A. (2011) · Annual Review of Psychology · doi:10.1146/annurev-psych-120709-145406 · Cited by 950
The Simulation of Smiles (SIMS) model: Embodied simulation and the meaning of facial expression
Niedenthal, P. M., Mermillod, M., Maringer, M., & Hess, U. (2010) · Behavioral and Brain Sciences · doi:10.1017/s0140525x10000865 · Cited by 618
Metadata
- Authors
- Dick, A. S., Goldin‐Meadow, S., Hasson, U., Skipper, J. I., & Small, S. L.
- Venue
- Human Brain Mapping
- Publisher
- Wiley
- Published
- 2009-04-21
- Volume
- 30
- Issue
- 11
- Pages
- 3509-3526
- ISSN
- 1065-9471 (ISSN-L)
- 1097-0193
- OpenAlex
- W2105902013
- Cited by
- 153
- References
- 28
- FWCI
- 16.01
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.