Interactions between Auditory and Visual Semantic Stimulus Classes: Evidence for Common Processing Networks for Speech and Body Actions
Open accessThe DOI matches its Crossref record; 18 of the 18 references it cites were found in a registry or an index.
Abstract
Incongruencies between auditory and visual signals negatively affect human performance and cause selective activation in neuroimaging studies; therefore, they are increasingly used to probe audiovisual integration mechanisms. An open question is whether the increased BOLD response reflects computational demands in integrating mismatching low-level signals or reflects simultaneous unimodal conceptual representations of the competing signals. To address this question, we explore the effect of semantic congruency within and across three signal categories (speech, body actions, and unfamiliar patterns) for signals with matched low-level statistics. In a localizer experiment, unimodal (auditory and visual) and bimodal stimuli were used to identify ROIs. All three semantic categories cause overlapping activation patterns. We find no evidence for areas that show greater BOLD response to bimodal stimuli than predicted by the sum of the two unimodal responses. Conjunction analysis of the unimodal responses in each category identifies a network including posterior temporal, inferior frontal, and premotor areas. Semantic congruency effects are measured in the main experiment. We find that incongruent combinations of two meaningful stimuli (speech and body actions) but not combinations of meaningful with meaningless stimuli lead to increased BOLD response in the posterior STS (pSTS) bilaterally, the left SMA, the inferior frontal gyrus, the inferior parietal lobule, and the anterior insula. These interactions are not seen in premotor areas. Our findings are consistent with the hypothesis that pSTS and frontal areas form a recognition network that combines sensory categorical representations (in pSTS) with action hypothesis generation in inferior frontal gyrus/premotor areas. We argue that the same neural networks process speech and body actions.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The episodic buffer: a new component of working memory?
Baddeley, A. (2000) · Trends in Cognitive Sciences · doi:10.1016/s1364-6613(00)01538-2 · Cited by 7,359 · cited 1×
“, Baddeley, 2000). Baddeley (2000) proposed three working memory buffers under the control of a central executive system.”
BackgroundRizzolatti, G., & Arbib, M. A. (1998) · Trends in Neurosciences · doi:10.1016/s0166-2236(98)01260-0 · Cited by 2,858
Hickok, G., & Poeppel, D. (2004) · Cognition · doi:10.1016/j.cognition.2003.10.011 · Cited by 2,212 · cited 1×
“Although the speech-related activation of the motor circuit is consistent with other findings (e.g., Skipper et al., 2005) and models (e.g., Hickok & Poeppel, 2004), one might have expected to also see some activation in action observation networks for the body action stimuli (Saygin, 2007).”
ResultBrain mechanisms linking language and action
Pulvermüller, F. (2005) · Nature Reviews Neuroscience · doi:10.1038/nrn1706 · Cited by 1,828 · cited 3×
“The pSTS demarcates the lower bank of Wernickeʼs area so that there is a wealth of literature linking speech perception deficits to lesions at this site (for a review, see, e.g., Damasio & Geschwind, 1984; but see Pulvermüller, 2005).”
BackgroundResultHearing Sounds, Understanding Actions: Action Representation in Mirror Neurons
Kohler, E., Keysers, C., Umiltà, M. A., Fogassi, L., Gallese, V., & Rizzolatti, G. (2002) · Science · doi:10.1126/science.1070311 · Cited by 1,801 · cited 1×
“Although the response is selective for specific actions, the modality of observation is not critical (Keysers et al., 2003; Kohler et al., 2002).”
BackgroundCrossmodal Processing in the Human Brain: Insights from Functional Neuroimaging Studies
Calvert, G. A. (2001) · Cerebral Cortex · doi:10.1093/cercor/11.12.1110 · Cited by 1,012
Calvert, G. A., Campbell, R., & Brammer, M. J. (2000) · Current Biology · doi:10.1016/s0960-9822(00)00513-3 · Cited by 954
Listening to speech activates motor areas involved in speech production
Wilson, S. M., Saygin, A. P., Sereno, M. I., & Iacoboni, M. (2004) · Nature Neuroscience · doi:10.1038/nn1263 · Cited by 885
Electrophysiology and brain imaging of biological motion
Puce, A., & Perrett, D. (2003) · Philosophical Transactions of the Royal Society of London. Series B: Biological Sciences · doi:10.1098/rstb.2002.1221 · Cited by 693
Friston, K. J., Penny, W. D., & Glaser, D. E. (2005) · NeuroImage · doi:10.1016/j.neuroimage.2005.01.013 · Cited by 578 · cited 1×
“…scan was designed to identify areas that show selective activation to bimodal compared with unimodal stimuli (audiovisual [AV] > auditory [A] + visual [V]) or to identify areas that are significantly activated by visual and auditory stimuli (a conjunction analysis: (A > Rest) ∩ (V > Rest)).”
BackgroundSeparate neural subsystems within ‘Wernicke's area’
Wise, R. J. S., Scott, S. K., Blank, S. C., Mummery, C. J., Murphy, K., & Warburton, E. A. (2001) · Brain · doi:10.1093/brain/124.1.83 · Cited by 572 · cited 2×
“This view is consistent with the suggestion that the pSTS may be a phonological buffer (Wise et al., 2001) and more recent proposals by Jacquemot and Scott (2006), who propose a role of the pSTS in short-term perceptual memory.”
ResultUnraveling multisensory integration: patchy organization within human STS multisensory cortex
Beauchamp, M. S., Argall, B. D., Bodurka, J., Duyn, J. H., & Martin, A. (2004) · Nature Neuroscience · doi:10.1038/nn1333 · Cited by 513 · cited 1×
“Superadditive effects in the posterior STS (pSTS) are also the basis for the claim that this structure is an audiovisual binding site (eg, Beauchamp, Argall, Bodurka, Duyn, & Martin, 2004 ; Calvert, 2001 ), which may not require semantics at all.”
BackgroundSuperior Temporal Sulcus—It's My Area: Or Is It?
Hein, G., & Knight, R. T. (2008) · Journal of Cognitive Neuroscience · doi:10.1162/jocn.2008.20148 · Cited by 486 · cited 2×
“Superior temporal areas, among them the pSTS, are also key speech areas and respond strongly to heard (for reviews, see, e.g., Hein & Knight, 2008; Hickok & Poeppel, 2007; Cabeza&Nyberg,2000),seen(e.g.,Skipper,Nusbaum, & Small, 2005), read (e.g., Cabeza & Nyberg, 2000), and subvocallyreproduced(Hickok&Buchsbaum,2003)s…”
BackgroundAudiovisual mirror neurons and action recognition
Keysers, C., Kohler, E., Umilt�, M. A., Nanetti, L., Fogassi, L., & Gallese, V. (2003) · Experimental Brain Research · doi:10.1007/s00221-003-1603-5 · Cited by 420
Pelphrey, K. A., Morris, J. P., Michelich, C. R., Allison, T., & McCarthy, G. (2005) · Cerebral Cortex · doi:10.1093/cercor/bhi064 · Cited by 369 · cited 1×
“This finding is mirrored by action observation studies using natural video sequences (eg, Pelphrey, Morris, Michelich, Allison, & McCarthy, 2005 ; Hamzei et al, 2003 ) that show differential activation in effector-specific sectors of Broca's area and premotor cortex (Buccino, Binkowski, & Riggio, 2004 ).”
ResultBrain activity evoked by inverted and imagined biological motion
Grossman, E. D., & Blake, R. (2001) · Vision Research · doi:10.1016/s0042-6989(00)00317-5 · Cited by 333 · cited 1×
“…2003; Vaina, Solomon, Chowdhury, Sinha, & Belliveau, 2001; Grossman et al., 2000), auditory (Bidet-Caulet, Voisin, Bertrand,& Fonlup, 2005; Pizzamiglio et al., 2005), audiovisual (e.g., Stevenson, Kim, & James, 2009; Beauchamp, 2005), and imagined (Grossman & Blake, 2001) biological motion signals.”
Damasio, A. R., & Geschwind, N. (1984) · Annual Review of Neuroscience · doi:10.1146/annurev.ne.07.030184.001015 · Cited by 298
Scott, S. K., McGettigan, C., & Eisner, F. (2009) · Nature Reviews Neuroscience · doi:10.1038/nrn2603 · Cited by 251 · cited 1×
“Most recent studies identify the STS as the site of prelexical speech categorization (for reviews, see Obleser & Eisner, 2009 ; Scott, McGettigan, & Eisner, 2009 ; Hickok & Poeppel, 2007 ).”
Background
Cited by
Diffusion tensor imaging shows white matter tracts between human auditory and visual cortex
Beer, A. L., Plank, T., & Greenlee, M. W. (2011) · Experimental Brain Research · doi:10.1007/s00221-011-2715-y · Cited by 124
Self-citationAn fMRI investigation of the relationship between future imagination and cognitive flexibility
Roberts, R. P., Wiebels, K., Sumner, R. L., van Mulukom, V., Grady, C. L., Schacter, D. L., & Addis, D. R. (2017) · Neuropsychologia · doi:10.1016/j.neuropsychologia.2016.11.019 · Cited by 58
Beer, A. L., Plank, T., Meyer, G., & Greenlee, M. W. (2013) · Frontiers in Integrative Neuroscience · doi:10.3389/fnint.2013.00005 · Cited by 53
“Several studies showing multisensory responses in the STS also reported multisensory activity in the inferior occipito-temporal cortex, anterior insula, and ventrolateral frontal cortex (Calvert et al., 2000; Beauchamp et al., 2004b; Hein et al., 2007; Meyer et al., 2011; Nath and Beauchamp, 2012).”
BackgroundMethodResultInfluentialNaumer, M. J., van den Bosch, J. J. F., Wibral, M., Kohler, A., Singer, W., Kaiser, J., van de Ven, V., & Muckli, L. (2011) · Experimental Brain Research · doi:10.1007/s00221-011-2669-0 · Cited by 11
Metadata
- Authors
- Meyer, G. F., Greenlee, M., & Wuerger, S.
- Venue
- Journal of Cognitive Neuroscience
- Publisher
- MIT Press - Journals
- Published
- 2011-09-01
- Volume
- 23
- Issue
- 9
- Pages
- 2291-2308
- ISSN
- 0898-929X (ISSN-L)
- 1530-8898
- OpenAlex
- W2127431490
- Cited by
- 52
- References
- 18
- FWCI
- 6.39
- Type
- journal article
- Language
- en
- Open access
- bronze
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.