Physical and Perceptual Factors Shape the Neural Mechanisms That Integrate Audiovisual Signals in Speech Comprehension
Open accessThe DOI matches its Crossref record; 20 of the 20 references it cites were found in a registry or an index.
Abstract
Face-to-face communication challenges the human brain to integrate information from auditory and visual senses with linguistic representations. Yet the role of bottom-up physical (spectrotemporal structure) input and top-down linguistic constraints in shaping the neural mechanisms specialized for integrating audiovisual speech signals are currently unknown. Participants were presented with speech and sinewave speech analogs in visual, auditory, and audiovisual modalities. Before the fMRI study, they were trained to perceive physically identical sinewave speech analogs as speech (SWS-S) or nonspeech (SWS-N). Comparing audiovisual integration (interactions) of speech, SWS-S, and SWS-N revealed a posterior–anterior processing gradient within the left superior temporal sulcus/gyrus (STS/STG): Bilateral posterior STS/STG integrated audiovisual inputs regardless of spectrotemporal structure or speech percept; in left mid-STS, the integration profile was primarily determined by the spectrotemporal structure of the signals; more anterior STS regions discarded spectrotemporal structure and integrated audiovisual signals constrained by stimulus intelligibility and the availability of linguistic representations. In addition to this “ventral” processing stream, a “dorsal” circuitry encompassing posterior STS/STG and left inferior frontal gyrus differentially integrated audiovisual speech and SWS signals. Indeed, dynamic causal modeling and Bayesian model comparison provided strong evidence for a parallel processing structure encompassing a ventral and a dorsal stream with speech intelligibility training enhancing the connectivity between posterior and anterior STS/STG. In conclusion, audiovisual speech comprehension emerges in an interactive process with the integration of auditory and visual signals being progressively constrained by stimulus intelligibility along the STS and spectrotemporal structure in a dorsal fronto-temporal circuitry.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The free-energy principle: a unified brain theory?
Friston, K. (2010) · Nature Reviews Neuroscience · doi:10.1038/nrn2787 · Cited by 8,120 · cited 1×
“In conclusion, our results suggest that both bottom-up spectrotemporal inputs and the availability of top-down linguistic (and articulatory) representations shape the neural processes underlying AV integration in speech perception (Davis and Johnsrude, 2007; Friston, 2009, 2010).”
The free-energy principle: a rough guide to the brain?
Friston, K. (2009) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2009.04.005 · Cited by 1,831
Hearing Sounds, Understanding Actions: Action Representation in Mirror Neurons
Kohler, E., Keysers, C., Umiltà, M. A., Fogassi, L., Gallese, V., & Rizzolatti, G. (2002) · Science · doi:10.1126/science.1070311 · Cited by 1,801 · cited 1×
“…premotor cortex (BA44)] forms part of the mirror neuron system (Nelissen et al., 2005; Petrides et al., 2005) that responds, when participants make or observe a particular action (Rizzolatti and Craighero, 2004; Fadiga et al., 2005) or perceive sounds produced by that action (Kohler et al., 2002).”
BackgroundMaps and streams in the auditory cortex: nonhuman primates illuminate human speech processing
Rauschecker, J. P., & Scott, S. K. (2009) · Nature Neuroscience · doi:10.1038/nn.2331 · Cited by 1,766
Human temporal lobe activation by speech and nonspeech sounds
Binder, J. R. (2000) · Cerebral Cortex · doi:10.1093/cercor/10.5.512 · Cited by 1,345 · cited 1×
“This invariance of the AV integration profile to physical stimulus features in the anterior portion of left mid-STS converges with previous findings in the auditory domain (Binder et al., 2000; Scott et al., 2000; Davis and Johnsrude, 2003; Davis et al., 2007).”
ResultListening to speech activates motor areas involved in speech production
Wilson, S. M., Saygin, A. P., Sereno, M. I., & Iacoboni, M. (2004) · Nature Neuroscience · doi:10.1038/nn1263 · Cited by 885 · cited 1×
“Given the dominant links between speech perception and production, a role in speech perception has also been invoked for articulatory-gestural representations (Liberman and Mattingly, 1985; Wilson et al., 2004; Skipper et al., 2007).”
BackgroundComparing dynamic causal models.
Penny, W. D., Stephan, K. E., Mechelli, A., & Friston, K. J. (2004) · NeuroImage · doi:10.1016/j.neuroimage.2004.03.026 · Cited by 848 · cited 2×
“To determine the most likely of the three DCMs given the observed data from all subjects, we implemented fixed- and random-effects group analyses (Penny et al., 2004, 2010).”
MethodTen simple rules for dynamic causal modeling
Stephan, K. E., Penny, W. D., Moran, R. J., den Ouden, H. E. M., Daunizeau, J., & Friston, K. J. (2010) · NeuroImage · doi:10.1016/j.neuroimage.2009.11.015 · Cited by 803 · cited 1×
“This is because the aim of DCM is to investigate how regionally specific effects emerge from interactions among brain areas, and therefore it is not advised to select brain regions that do not show significant activations (Stephan et al., 2010).”
BackgroundComparing families of dynamic causal models
Penny, W. D., Stephan, K. E., Daunizeau, J., Rosa, M. J., Friston, K. J., Schofield, T. M., & Leff, A. P. (2010) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1000709 · Cited by 688 · cited 3×
“…at the random-effects level, we report (1) the expectation of the posterior probability (i.e., the expected likelihood of obtaining the kth model for any randomly selected subject) and (2) the exceedance probability of one model being more likely than any other model tested (Penny et al., 2010).”
BackgroundMethodFriston, K. J., Penny, W. D., & Glaser, D. E. (2005) · NeuroImage · doi:10.1016/j.neuroimage.2005.01.013 · Cited by 578 · cited 1×
“The left posterior STS/STG was only marginally significant at the statistical threshold of 0.07 using the very stringent and overly conservative conjunction null conjunction analysis (Friston et al., 2005).”
MethodInfluentialUnraveling multisensory integration: patchy organization within human STS multisensory cortex
Beauchamp, M. S., Argall, B. D., Bodurka, J., Duyn, J. H., & Martin, A. (2004) · Nature Neuroscience · doi:10.1038/nn1333 · Cited by 513 · cited 5×
“…AV integration profile common to all stimulus classes regardless of their spectrotemporal structure or speech intelligibility (Wright et al., 2003; Beauchamp et al., 2004, 2008; van Atteveldt et al., 2004; Amedi et al., 2005; Barraclough et al., 2005; Beauchamp, 2005; Miller and D’Esposito, 2005;…”
BackgroundInfluentialObserving others: multiple action representation in the frontal lobe.
Nelissen, K., Luppino, G., Vanduffel, W., Rizzolatti, G., & Orban, G. A. (2005) · Science · doi:10.1126/science.1115593 · Cited by 352
Orofacial somatomotor responses in the macaque monkey homologue of Broca's area
Petrides, M., Cadoret, G., & Mackey, S. (2005) · Nature · doi:10.1038/nature03628 · Cited by 337
Converging language streams in the human temporal lobe
Spitsyna, G., Warren, J. E., Scott, S. K., Turkheimer, F. E., & Wise, R. J. S. (2006) · The Journal of Neuroscience · doi:10.1523/jneurosci.0559-06.2006 · Cited by 312 · cited 1×
“Anterior temporal areas have also been shown to be activated by both written and spoken words indicating a role in amodal lexical and semantic access (Spitsyna et al., 2006).”
BackgroundImplicit multisensory associations influence voice recognition
von Kriegstein, K., & Giraud, A.-L. (2006) · PLoS Biology · doi:10.1371/journal.pbio.0040326 · Cited by 309 · cited 1×
“Interestingly, the BOLD response elicited by “visual” facial movements was modulated by prior cross-modal learning (Gonzalo et al., 2000; von Kriegstein and Giraud, 2006).”
BackgroundDual Neural Routing of Visual Facilitation in Speech Processing
Arnal, L. H., Morillon, B., Kell, C. A., & Giraud, A.-L. (2009) · The Journal of Neuroscience · doi:10.1523/jneurosci.3194-09.2009 · Cited by 241
Touch, sound and vision in human superior temporal sulcus
Beauchamp, M. S., Yasar, N. E., Frye, R. E., & Ro, T. (2008) · NeuroImage · doi:10.1016/j.neuroimage.2008.03.015 · Cited by 208
Möttönen, R., Calvert, G. A., Jääskeläinen, I. P., Matthews, P. M., Thesen, T., Tuomainen, J., & Sams, M. (2006) · NeuroImage · doi:10.1016/j.neuroimage.2005.10.002 · Cited by 158 · cited 1×
“Stimuli included AV, auditory (A), and visual (V) signals of speech and SWS (Remez et al., 1983; Tuomainen et al., 2005; Benson et al., 2006; Möttönen et al., 2006; Desai et al., 2008).”
BackgroundDynamic causal modeling: a generative model of slice timing in fMRI
Kiebel, S. J., Klöppel, S., Weiskopf, N., & Friston, K. J. (2007) · NeuroImage · doi:10.1016/j.neuroimage.2006.10.026 · Cited by 92 · cited 2×
“To transform auditory speech into sinewave speech analogs, the audio tracks were separated from the video tracks.”
BackgroundMethodPhonetic processing areas revealed by sinewave speech and acoustically similar non-speech
Benson, R. R., Richardson, M., Whalen, D. H., & Lai, S. (2006) · NeuroImage · doi:10.1016/j.neuroimage.2005.11.029 · Cited by 50
Cited by
Expecting to See a Letter: Alpha Oscillations as Carriers of Top-Down Sensory Predictions
Mayer, A., Schwiedrzik, C. M., Wibral, M., Singer, W., & Melloni, L. (2016) · Cerebral Cortex · doi:10.1093/cercor/bhv146 · Cited by 119
Frontal cortex selects representations of the talker’s mouth to aid in speech perception
Ozker, M., Yoshor, D., & Beauchamp, M. S. (2018) · eLife · doi:10.7554/elife.30387 · Cited by 24
“DOI: https://doi.org/10.7554/eLife.30387 7 of 14 Hall et al., 2005; Lee and Noppeney, 2011; Okada and Hickok, 2009).”
BackgroundInfluentialAuditory Deficits in Audiovisual Speech Perception in Adult Asperger’s Syndrome: fMRI Study
Tietze, F.-A., Hundertmark, L., Roy, M., Zerr, M., Sinke, C., Wiswede, D., Walter, M., Münte, T. F., & Szycik, G. R. (2019) · Frontiers in Psychology · doi:10.3389/fpsyg.2019.02286 · Cited by 16
“In particular, the superior temporal sulcus (STS) typically displays increased activation for AV stimuli compared to unimodal presentation and also shows congruency effects (Calvert et al., 2000; Sekiyama et al., 2003; Wright et al., 2003; Brefczynski-Lewis et al., 2009; Szycik et al., 2009a, 2012; Lee and Noppeney, 2…”
Background
Metadata
- Authors
- Lee, H., & Noppeney, U.
- Venue
- The Journal of Neuroscience
- Publisher
- Society for Neuroscience
- Published
- 2011-08-03
- Volume
- 31
- Issue
- 31
- Pages
- 11338-11350
- ISSN
- 0270-6474 (ISSN-L)
- 1529-2401
- OpenAlex
- W2017221256
- Cited by
- 58
- References
- 20
- FWCI
- 6.72
- Type
- journal article
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.