Physical and Perceptual Factors Shape the Neural Mechanisms That Integrate Audiovisual Signals in Speech Comprehension
Open accessThe DOI matches its Crossref record; 17 of the 17 references it cites were found in a registry or an index.
Abstract
Face-to-face communication challenges the human brain to integrate information from auditory and visual senses with linguistic representations. Yet the role of bottom-up physical (spectrotemporal structure) input and top-down linguistic constraints in shaping the neural mechanisms specialized for integrating audiovisual speech signals are currently unknown. Participants were presented with speech and sinewave speech analogs in visual, auditory, and audiovisual modalities. Before the fMRI study, they were trained to perceive physically identical sinewave speech analogs as speech (SWS-S) or nonspeech (SWS-N). Comparing audiovisual integration (interactions) of speech, SWS-S, and SWS-N revealed a posterior–anterior processing gradient within the left superior temporal sulcus/gyrus (STS/STG): Bilateral posterior STS/STG integrated audiovisual inputs regardless of spectrotemporal structure or speech percept; in left mid-STS, the integration profile was primarily determined by the spectrotemporal structure of the signals; more anterior STS regions discarded spectrotemporal structure and integrated audiovisual signals constrained by stimulus intelligibility and the availability of linguistic representations. In addition to this “ventral” processing stream, a “dorsal” circuitry encompassing posterior STS/STG and left inferior frontal gyrus differentially integrated audiovisual speech and SWS signals. Indeed, dynamic causal modeling and Bayesian model comparison provided strong evidence for a parallel processing structure encompassing a ventral and a dorsal stream with speech intelligibility training enhancing the connectivity between posterior and anterior STS/STG. In conclusion, audiovisual speech comprehension emerges in an interactive process with the integration of auditory and visual signals being progressively constrained by stimulus intelligibility along the STS and spectrotemporal structure in a dorsal fronto-temporal circuitry.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The free-energy principle: a unified brain theory?
Friston, K. (2010) · Nature Reviews Neuroscience · doi:10.1038/nrn2787 · Cited by 8,120 · cited 1×
“In conclusion, our results suggest that both bottom-up spectrotemporal inputs and the availability of top-down linguistic (and articulatory) representations shape the neural processes underlying AV integration in speech perception (Davis and Johnsrude, 2007; Friston, 2009, 2010).”
The free-energy principle: a rough guide to the brain?
Friston, K. (2009) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2009.04.005 · Cited by 1,831
Hearing Sounds, Understanding Actions: Action Representation in Mirror Neurons
Kohler, E., Keysers, C., Umiltà, M. A., Fogassi, L., Gallese, V., & Rizzolatti, G. (2002) · Science · doi:10.1126/science.1070311 · Cited by 1,801 · cited 1×
“…premotor cortex (BA44)] forms part of the mirror neuron system (Nelissen et al., 2005; Petrides et al., 2005) that responds, when participants make or observe a particular action (Rizzolatti and Craighero, 2004; Fadiga et al., 2005) or perceive sounds produced by that action (Kohler et al., 2002).”
BackgroundHuman temporal lobe activation by speech and nonspeech sounds
Binder, J. R. (2000) · Cerebral Cortex · doi:10.1093/cercor/10.5.512 · Cited by 1,345 · cited 1×
“This invariance of the AV integration profile to physical stimulus features in the anterior portion of left mid-STS converges with previous findings in the auditory domain (Binder et al., 2000; Scott et al., 2000; Davis and Johnsrude, 2003; Davis et al., 2007).”
ResultListening to speech activates motor areas involved in speech production
Wilson, S. M., Saygin, A. P., Sereno, M. I., & Iacoboni, M. (2004) · Nature Neuroscience · doi:10.1038/nn1263 · Cited by 885 · cited 1×
“Given the dominant links between speech perception and production, a role in speech perception has also been invoked for articulatory-gestural representations (Liberman and Mattingly, 1985; Wilson et al., 2004; Skipper et al., 2007).”
BackgroundComparing dynamic causal models.
Penny, W. D., Stephan, K. E., Mechelli, A., & Friston, K. J. (2004) · NeuroImage · doi:10.1016/j.neuroimage.2004.03.026 · Cited by 848 · cited 2×
“To determine the most likely of the three DCMs given the observed data from all subjects, we implemented fixed- and random-effects group analyses (Penny et al., 2004, 2010).”
MethodTen simple rules for dynamic causal modeling
Stephan, K. E., Penny, W. D., Moran, R. J., den Ouden, H. E. M., Daunizeau, J., & Friston, K. J. (2010) · NeuroImage · doi:10.1016/j.neuroimage.2009.11.015 · Cited by 803 · cited 1×
“This is because the aim of DCM is to investigate how regionally specific effects emerge from interactions among brain areas, and therefore it is not advised to select brain regions that do not show significant activations (Stephan et al., 2010).”
BackgroundComparing families of dynamic causal models
Penny, W. D., Stephan, K. E., Daunizeau, J., Rosa, M. J., Friston, K. J., Schofield, T. M., & Leff, A. P. (2010) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1000709 · Cited by 688 · cited 3×
“…at the random-effects level, we report (1) the expectation of the posterior probability (i.e., the expected likelihood of obtaining the kth model for any randomly selected subject) and (2) the exceedance probability of one model being more likely than any other model tested (Penny et al., 2010).”
BackgroundMethodUnraveling multisensory integration: patchy organization within human STS multisensory cortex
Beauchamp, M. S., Argall, B. D., Bodurka, J., Duyn, J. H., & Martin, A. (2004) · Nature Neuroscience · doi:10.1038/nn1333 · Cited by 513 · cited 5×
“…AV integration profile common to all stimulus classes regardless of their spectrotemporal structure or speech intelligibility (Wright et al., 2003; Beauchamp et al., 2004, 2008; van Atteveldt et al., 2004; Amedi et al., 2005; Barraclough et al., 2005; Beauchamp, 2005; Miller and D’Esposito, 2005;…”
BackgroundInfluentialObserving others: multiple action representation in the frontal lobe.
Nelissen, K., Luppino, G., Vanduffel, W., Rizzolatti, G., & Orban, G. A. (2005) · Science · doi:10.1126/science.1115593 · Cited by 352
Cited by
Expecting to See a Letter: Alpha Oscillations as Carriers of Top-Down Sensory Predictions
Mayer, A., Schwiedrzik, C. M., Wibral, M., Singer, W., & Melloni, L. (2016) · Cerebral Cortex · doi:10.1093/cercor/bhv146 · Cited by 119
Frontal cortex selects representations of the talker’s mouth to aid in speech perception
Ozker, M., Yoshor, D., & Beauchamp, M. S. (2018) · eLife · doi:10.7554/elife.30387 · Cited by 24
“DOI: https://doi.org/10.7554/eLife.30387 7 of 14 Hall et al., 2005; Lee and Noppeney, 2011; Okada and Hickok, 2009).”
BackgroundInfluentialAuditory Deficits in Audiovisual Speech Perception in Adult Asperger’s Syndrome: fMRI Study
Tietze, F.-A., Hundertmark, L., Roy, M., Zerr, M., Sinke, C., Wiswede, D., Walter, M., Münte, T. F., & Szycik, G. R. (2019) · Frontiers in Psychology · doi:10.3389/fpsyg.2019.02286 · Cited by 16
“In particular, the superior temporal sulcus (STS) typically displays increased activation for AV stimuli compared to unimodal presentation and also shows congruency effects (Calvert et al., 2000; Sekiyama et al., 2003; Wright et al., 2003; Brefczynski-Lewis et al., 2009; Szycik et al., 2009a, 2012; Lee and Noppeney, 2…”
Background
Metadata
- Authors
- Lee, H., & Noppeney, U.
- Venue
- The Journal of Neuroscience
- Publisher
- Society for Neuroscience
- Published
- 2011-08-03
- Volume
- 31
- Issue
- 31
- Pages
- 11338-11350
- ISSN
- 0270-6474 (ISSN-L)
- 1529-2401
- OpenAlex
- W2017221256
- Cited by
- 58
- References
- 17
- FWCI
- 6.72
- Type
- journal article
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Physical and Perceptual Factors Shape the Neural Mechanisms That Integrate Audiovisual Signals in Speech Comprehension. https://app.cochat.ai/papers/10.1523/jneurosci.6510-10.2011
- The paper
- Lee, H., & Noppeney, U. (2011). Physical and Perceptual Factors Shape the Neural Mechanisms That Integrate Audiovisual Signals in Speech Comprehension. The Journal of Neuroscience. https://doi.org/10.1523/jneurosci.6510-10.2011
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.