Congruent Visual Speech Enhances Cortical Entrainment to Continuous Auditory Speech in Noise-Free Conditions
Open accessThe DOI matches its Crossref record; 67 of the 67 references it cites were found in a registry or an index.
Abstract
Congruent audiovisual speech enhances our ability to comprehend a speaker, even in noise-free conditions. When incongruent auditory and visual information is presented concurrently, it can hinder a listener's perception and even cause him or her to perceive information that was not presented in either modality. Efforts to investigate the neural basis of these effects have often focused on the special case of discrete audiovisual syllables that are spatially and temporally congruent, with less work done on the case of natural, continuous speech. Recent electrophysiological studies have demonstrated that cortical response measures to continuous auditory speech can be easily obtained using multivariate analysis methods. Here, we apply such methods to the case of audiovisual speech and, importantly, present a novel framework for indexing multisensory integration in the context of continuous speech. Specifically, we examine how the temporal and contextual congruency of ongoing audiovisual speech affects the cortical encoding of the speech envelope in humans using electroencephalography. We demonstrate that the cortical representation of the speech envelope is enhanced by the presentation of congruent audiovisual speech in noise-free conditions. Furthermore, we show that this is likely attributable to the contribution of neural generators that are not particularly active during unimodal stimulation and that it is most prominent at the temporal scale corresponding to syllabic rate (2–6 Hz). Finally, our data suggest that neural entrainment to the speech envelope is inhibited when the auditory and visual streams are incongruent both temporally and contextually. SIGNIFICANCE STATEMENT Seeing a speaker's face as he or she talks can greatly help in understanding what the speaker is saying. This is because the speaker's facial movements relay information about what the speaker is saying, but also, importantly, when the speaker is saying it. Studying how the brain uses this timing relationship to combine information from continuous auditory and visual speech has traditionally been methodologically difficult. Here we introduce a new approach for doing this using relatively inexpensive and noninvasive scalp recordings. Specifically, we show that the brain's representation of auditory speech is enhanced when the accompanying visual speech signal shares the same timing. Furthermore, we show that this enhancement is most pronounced at a time scale that corresponds to mean syllable length.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Delorme, A., & Makeig, S. (2004) · Journal of Neuroscience Methods · doi:10.1016/j.jneumeth.2003.10.009 · Cited by 24,371 · cited 1×
“Channels contaminated by noise were recalculated by spline-interpolating the surrounding clean channels in EEGLAB (Delorme and Makeig, 2004).”
MethodA tutorial on hidden Markov models and selected applications in speech recognition
Rabiner, L. R. (1989) · Proceedings of the IEEE · doi:10.1109/5.18626 · Cited by 22,743 · cited 2×
“In addition, using other approaches to quantify AV correlations such as those based on mutual information models (Nock et al., 2002) and hidden Markov models (Rabiner, 1989) may provide important complementary insights.”
Hearing lips and seeing voices
MCGURK, H., & MACDONALD, J. (1976) · Nature · doi:10.1038/264746a0 · Cited by 6,183 · cited 1×
“This is unsurprising, given that particular combinations of incongruent AV syllables elicit illusory percepts when presented concurrently (McGurk and MacDonald, 1976).”
BackgroundKruskal, J., & Wish, M. (1978) · Sage eBooks · doi:10.4135/9781412985130 · Cited by 3,751
Visual Contribution to Speech Intelligibility in Noise
Sumby, W. H., & Pollack, I. (1954) · The Journal of the Acoustical Society of America · doi:10.1121/1.1907309 · Cited by 2,583 · cited 2×
“Future work examining this using a speech-in-noise paradigm should prove more informative given the well-established benefits of multisensory speech in adverse hearing conditions (Sumby and Pollack, 1954; Ross et al., 2007a).”
BackgroundCortical oscillations and speech processing: emerging computational principles and operations
Giraud, A.-L., & Poeppel, D. (2012) · Nature Neuroscience · doi:10.1038/nn.3063 · Cited by 1,917 · cited 2×
“…framework, we also investigated how our measures of multisensory interaction varied across different temporal scales with a view to elucidating whether the effects were more prominent at any particular level of speech processing (i.e., phonemic, syllabic, word, prosodic; Giraud and Poeppel, 2012).”
Background(1994) · Choice Reviews Online · doi:10.5860/choice.31-4365 · Cited by 1,882
On the interpretation of weight vectors of linear models in multivariate neuroimaging
Haufe, S., Meinecke, F., Görgen, K., Dähne, S., Haynes, J.-D., Blankertz, B., & Bießmann, F. (2014) · NeuroImage · doi:10.1016/j.neuroimage.2013.10.067 · Cited by 1,389 · cited 4×
“, nonzero weights are only observed at channels where cortical activity is related to stimulus encoding (Haufe et al., 2014).”
BackgroundInfluentialDivided attention: Evidence for coactivation with redundant signals
Miller, J. (1982) · Cognitive Psychology · doi:10.1016/0010-0285(82)90010-x · Cited by 1,213 · cited 1×
“Violation of the race model was examined using the following inequality (Miller, 1982): FAVc t FA t FV t , t 0, (1) where FAVc, FA, and FV are the cumulative distribution functions (CDFs) based on the RTs of the AVc, A, and V conditions, respectively.”
MethodImmediate perceptual response to intersensory discrepancy.
Welch, R. B., & Warren, D. H. (1980) · Psychological Bulletin · doi:10.1037/0033-2909.88.3.638 · Cited by 1,176 · cited 1×
“This notion also fits with the theory that during conflicting AV presentation such as the McGurk scenario, directing attention toward a particular modality tends to reduce the bias of the unattended modality (Welch and Warren, 1980; Talsma et al., 2010).”
ResultTemporal information in speech: acoustic, auditory and linguistic aspects
Rosen, S. (1992) · Philosophical Transactions of the Royal Society of London. Series B: Biological Sciences · doi:10.1098/rstb.1992.0070 · Cited by 1,130
Visual speech speeds up the neural processing of auditory speech
van Wassenhove, V., Grant, K. W., & Poeppel, D. (2005) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.0408949102 · Cited by 854 · cited 1×
“Indeed, many EEG/MEG studies have reported interesting cross-modal interaction effects on cortical response measures, even when the discrete stimuli were phonetically incongruent (Klucharev et al., 2003; van Wassenhove et al., 2005; Stekelenburg and Vroomen, 2007; Arnal et al., 2009).”
BackgroundEmergence of neural encoding of auditory objects while listening to competing speakers
Ding, N., & Simon, J. Z. (2012) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1205381109 · Cited by 835 · cited 7×
“…and theoretical perspectives on stream segregation suggest that multifeature auditory sources are segregated into perceptual objects based on the temporal coherence of the neuronal responses to the various acoustic features (Elhilali et al., 2009; Shamma et al., 2011; Ding and Simon, 2012).”
BackgroundInfluentialThe multifaceted interplay between attention and multisensory integration.
Talsma, D., Senkowski, D., Soto-Faraco, S., & Woldorff, M. G. (2010) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2010.06.008 · Cited by 828 · cited 1×
“This notion also fits with the theory that during conflicting AV presentation such as the McGurk scenario, directing attention toward a particular modality tends to reduce the bias of the unattended modality (Welch and Warren, 1980; Talsma et al., 2010).”
ResultAttentional Selection in a Cocktail Party Environment Can Be Decoded from Single-Trial EEG
O'Sullivan, J. A., Power, A. J., Mesgarani, N., Rajaram, S., Foxe, J. J., Shinn-Cunningham, B. G., Slaney, M., Shamma, S. A., & Lalor, E. C. (2015) · Cerebral Cortex · doi:10.1093/cercor/bht355 · Cited by 824
Self-citationIntegration of Auditory and Visual Information about Objects in Superior Temporal Sulcus
Beauchamp, M. S., Lee, K. E., Argall, B. D., & Martin, A. (2004) · Neuron · doi:10.1016/s0896-6273(04)00070-4 · Cited by 773 · cited 1×
“…populations might facilitate the binding of temporally coherent visual and auditory speech, one candidate region is the superior temporal sulcus, which has previously been linked with multisensory object formation (Calvert and Campbell, 2003; Beauchamp et al., 2004; Kayser and Logothetis, 2009).”
BackgroundDIVISION OF PSYCHOLOGY : STATISTICAL FACILITATION OF SIMPLE REACTION TIMES*
Raab, D. H. (1962) · Transactions of the New York Academy of Sciences · doi:10.1111/j.2164-0947.1962.tb01433.x · Cited by 668 · cited 1×
“An RSE does not necessarily imply multisensory interaction unless it violates the race model (Raab, 1962).”
BackgroundRoss, L. A., Saint-Amour, D., Leavitt, V. M., Javitt, D. C., & Foxe, J. J. (2006) · Cerebral Cortex · doi:10.1093/cercor/bhl024 · Cited by 650
The Natural Statistics of Audiovisual Speech
Chandrasekaran, C., Trubanova, A., Stillittano, S., Caplier, A., & Ghazanfar, A. A. (2009) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1000436 · Cited by 630 · cited 4×
“The uppermost and lowermost filter limits captured the first, second, and third formant spectral regions of the speech signals, known to carry the acoustic information that correlates most with visual speech features (Grant and Seitz, 2000; Chandrasekaran et al., 2009).”
BackgroundResultInfluentialReconstructing Speech from Human Auditory Cortex
Pasley, B. N., David, S. V., Mesgarani, N., Flinker, A., Shamma, S. A., Crone, N. E., Knight, R. T., & Chang, E. F. (2012) · PLoS Biology · doi:10.1371/journal.pbio.1001251 · Cited by 604 · cited 1×
“Our decision to include all 128 channels of EEG in the reconstruction analysis is justified because irrelevant filter channels can maintain zero weight while allowing the model to capture additional variance (Pasley et al., 2012).”
BackgroundCategorical speech representation in human superior temporal gyrus
Chang, E. F., Rieger, J. W., Johnson, K., Berger, M. S., Barbaro, N. M., & Knight, R. T. (2010) · Nature Neuroscience · doi:10.1038/nn.2641 · Cited by 588 · cited 1×
“MDS has been applied to electrophysiological data in previous studies to demonstrate the dissimilarity of neural responses elicited to different phonemes (Chang et al., 2010; Di Liberto et al., 2015).”
MethodLow-Frequency Cortical Entrainment to Speech Reflects Phoneme-Level Processing
Di Liberto, G. M., O’Sullivan, J. A., & Lalor, E. C. (2015) · Current Biology · doi:10.1016/j.cub.2015.08.030 · Cited by 578 · cited 2×
“This endeavor might be aided further by extending the framework to reduce the reliance on the acoustic envelope by directly incorporating information about phonemes and visemes as has been done recently for unimodal auditory speech (Di Liberto et al., 2015).”
BackgroundMethodSelf-citationSpeech comprehension is correlated with temporal response patterns recorded from auditory cortex
Ahissar, E., Nagarajan, S., Ahissar, M., Protopapas, A., Mahncke, H., & Merzenich, M. M. (2001) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.201400998 · Cited by 538 · cited 1×
“Electroencephalography (EEG) and magnetoencephalography (MEG) studies have demonstrated that auditory cortical activity entrains to the temporal envelope of speech (Ahissar et al., 2001; Abrams et al., 2008; Lalor and Foxe, 2010).”
BackgroundThe use of visible speech cues for improving auditory detection of spoken sentences
Grant, K. W., & Seitz, P.-F. (2000) · The Journal of the Acoustical Society of America · doi:10.1121/1.1288668 · Cited by 521 · cited 4×
“The uppermost and lowermost filter limits captured the first, second, and third formant spectral regions of the speech signals, known to carry the acoustic information that correlates most with visual speech features (Grant and Seitz, 2000; Chandrasekaran et al., 2009).”
BackgroundInfluentialVisual Modulation of Neurons in Auditory Cortex
Kayser, C., Petkov, C. I., & Logothetis, N. K. (2008) · Cerebral Cortex · doi:10.1093/cercor/bhm187 · Cited by 509 · cited 1×
“…articulatory movements) that may elicit a phase-reset of ongoing low-frequency oscillations in auditory cortex, such that the arrival of the corresponding auditory syllable coincides with a high excitability phase of the auditory neuronal population (Kayser et al., 2008; Schroeder et al., 2008).”
BackgroundNeuronal oscillations and visual amplification of speech
Schroeder, C. E., Lakatos, P., Kajikawa, Y., Partan, S., & Puce, A. (2008) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2008.01.002 · Cited by 498 · cited 3×
“…articulatory movements) that may elicit a phase-reset of ongoing low-frequency oscillations in auditory cortex, such that the arrival of the corresponding auditory syllable coincides with a high excitability phase of the auditory neuronal population (Kayser et al., 2008; Schroeder et al., 2008).”
BackgroundCortical entrainment to continuous speech: functional roles and interpretations
Ding, N., & Simon, J. Z. (2014) · Frontiers in Human Neuroscience · doi:10.3389/fnhum.2014.00311 · Cited by 443 · cited 3×
“…to EEG to determine the details of the neural mechanisms underlying AV speech integration, we suggest that the effects reported here are mediated by the temporal coherence of congruent AV speech at the syllabic time scale as part of an analysis-by-synthesis process (Ding and Simon, 2014).”
BackgroundMethodInfluentialTemporal coherence and attention in auditory scene analysis
Shamma, S. A., Elhilali, M., & Micheyl, C. (2011) · Trends in Neurosciences · doi:10.1016/j.tins.2010.11.002 · Cited by 429
Some preliminaries to a comprehensive account of audio-visual speech perception.
Summerfield, A. Q. (1987) · Cited by 421
Seeing speech: visual information from lip movements modifies activity in the human auditory cortex
Sams, M., Aulanko, R., Hämäläinen, M., Hari, R., Lounasmaa, O. V., Lu, S.-T., & Simola, J. (1991) · Neuroscience Letters · doi:10.1016/0304-3940(91)90914-f · Cited by 367 · cited 2×
“Traditionally, EEG/MEG studies have focused on how the brain responds to discrete AV stimuli such as syllables (Sams et al., 1991; Möttönen et al., 2002), an approach that is limited in what it can say about the role of the temporal correlation between continuous auditory and visual speech.”
BackgroundLipreading and audio-visual speech perception
Summerfield, Q. (1992) · Philosophical Transactions of the Royal Society of London. Series B: Biological Sciences · doi:10.1098/rstb.1992.0009 · Cited by 358
Grant, K. W., Walden, B. E., & Seitz, P. F. (1998) · The Journal of the Acoustical Society of America · doi:10.1121/1.422788 · Cited by 342 · cited 1×
“Visual speech assumes a complementary role when it is required to compensate for underspecified auditory speech, enhancing perception, e.g., in adverse hearing conditions (Sumby and Pollack, 1954; Ross et al., 2007a) and in people with impaired hearing (Grant et al., 1998).”
BackgroundNeural Correlates of Multisensory Integration of Ecologically Valid Audiovisual Events
Stekelenburg, J. J., & Vroomen, J. (2007) · Journal of Cognitive Neuroscience · doi:10.1162/jocn.2007.19.12.1964 · Cited by 336
Reading Speech from Still and Moving Faces: The Neural Substrates of Visible Speech
Calvert, G. A., & Campbell, R. (2003) · Journal of Cognitive Neuroscience · doi:10.1162/089892903321107828 · Cited by 332 · cited 2×
“…populations might facilitate the binding of temporally coherent visual and auditory speech, one candidate region is the superior temporal sulcus, which has previously been linked with multisensory object formation (Calvert and Campbell, 2003; Beauchamp et al., 2004; Kayser and Logothetis, 2009).”
BackgroundAdaptive Temporal Encoding Leads to a Background-Insensitive Cortical Representation of Speech
Ding, N., & Simon, J. Z. (2013) · The Journal of Neuroscience · doi:10.1523/jneurosci.5297-12.2013 · Cited by 323 · cited 2×
“Recently, Ding and Simon (2013) demonstrated that the accuracy with which the envelope can be reconstructed from MEG our multisensory effects [i.e., AVc (A V)] may be occurring on the time scale of phonemes, syllables, words, or sentences.”
BackgroundBimodal speech: early suppressive visual effects in human auditory cortex
Besle, J., Fort, A., Delpuech, C., & Giard, M. (2004) · European Journal of Neuroscience · doi:10.1111/j.1460-9568.2004.03670.x · Cited by 323 · cited 2×
“The validity of the additive model for the purpose of indexing multisensory integration in electrophysiological studies is well established (Besle et al., 2004a).”
BackgroundResult(1995) · Proceedings of the Royal Society of London. Series B: Biological Sciences · doi:10.1098/rspb.1995.0204 · Cited by 317 · cited 2×
“We also implement several follow-up experiments to answer the following research questions: (1) Is a dynamic human face sufficient to enhance envelope tracking, even when it is temporally incongruent? (2) Does contextually incongruent information, such as conflicting gender, modulate envelope tracking differently? (3)…”
MethodNeural responses to uninterrupted natural speech can be extracted with precise temporal resolution
Lalor, E. C., & Foxe, J. J. (2010) · European Journal of Neuroscience · doi:10.1111/j.1460-9568.2009.07055.x · Cited by 288 · cited 2×
“Electroencephalography (EEG) and magnetoencephalography (MEG) studies have demonstrated that auditory cortical activity entrains to the temporal envelope of speech (Ahissar et al., 2001; Abrams et al., 2008; Lalor and Foxe, 2010).”
BackgroundMethodSelf-citationRight-Hemisphere Auditory Cortex Is Dominant for Coding Syllable Patterns in Speech
Abrams, D. A., Nicol, T., Zecker, S., & Kraus, N. (2008) · The Journal of Neuroscience · doi:10.1523/jneurosci.0187-08.2008 · Cited by 282
The processing of audio-visual speech: empirical and neural bases
Campbell, R. (2008) · Philosophical Transactions of the Royal Society B: Biological Sciences · doi:10.1098/rstb.2007.2155 · Cited by 282 · cited 3×
“In keeping with previous work espousing a correlated mode of AV speech (Campbell, 2008), we postulate that visual speech features, being correlated with the envelope, results in the visual signal being bound to the auditory features to form a multisensory object.”
BackgroundVisual Input Enhances Selective Speech Envelope Tracking in Auditory Cortex at a “Cocktail Party”
Zion Golumbic, E., Cogan, G. B., Schroeder, C. E., & Poeppel, D. (2013) · The Journal of Neuroscience · doi:10.1523/jneurosci.3675-12.2013 · Cited by 258 · cited 4×
“It has been suggested that the integration of auditory and visual speech could be driven by the temporal coherence of cross-modal information (Zion Golumbic et al., 2013).”
BackgroundResultInfluentialPrediction and constraint in audiovisual speech perception
Peelle, J. E., & Sommers, M. S. (2015) · Cortex · doi:10.1016/j.cortex.2015.03.006 · Cited by 246 · cited 3×
“It has been suggested (Peelle and Sommers, 2015) that earlier integration effects are likely reflective of increased auditory cortical sensitivity to acoustic information; thus, we predict that they may be more evident in complementary modes of AV speech such as speech-in-noise.”
BackgroundRobust cortical entrainment to the speech envelope relies on the spectro-temporal fine structure
Ding, N., Chatterjee, M., & Simon, J. Z. (2014) · NeuroImage · doi:10.1016/j.neuroimage.2013.10.054 · Cited by 244 · cited 1×
“Recently, Ding et al. (2014) demonstrated that cortical entrainment to the speech envelope does not reflect encoding of the envelope per se, as it relies on the spectrotemporal fine structure of speech.”
BackgroundDual Neural Routing of Visual Facilitation in Speech Processing
Arnal, L. H., Morillon, B., Kell, C. A., & Giraud, A.-L. (2009) · The Journal of Neuroscience · doi:10.1523/jneurosci.3194-09.2009 · Cited by 241 · cited 5×
“, 2010), and fMRI work, which has demonstrated that the source of the visual facilitation of speech arises from motionsensitive cortex (Arnal et al., 2009).”
BackgroundInfluentialAt what time is the cocktail party? A late locus of selective attention to natural speech
Power, A. J., Foxe, J. J., Forde, E., Reilly, R. B., & Lalor, E. C. (2012) · European Journal of Neuroscience · doi:10.1111/j.1460-9568.2012.08060.x · Cited by 239 · cited 2×
“Although many studies have examined the effects of attention on envelope tracking (Ding and Simon, 2012; Power et al., 2012; Sheedy et al., 2014), less work has examined how this process may be influenced by visual speech [but see Zion Golumbic et al.”
BackgroundSelf-citationBrandwein, A. B., Foxe, J. J., Butler, J. S., Russo, N. N., Altschuler, T. S., Gomes, H., & Molholm, S. (2013) · Cerebral Cortex · doi:10.1093/cercor/bhs109 · Cited by 236 · cited 1×
“This naturalistic approach may prove useful in research with clinical populations in which altered multisensory (AV) processing has been reported, e.g., dyslexia (Hairston et al., 2005), autism (Brandwein et al., 2013), and schizophrenia (Ross et al., 2007b; Stekelenburg et al., 2013).”
BackgroundSelf-citationLuo, H., Liu, Z., & Poeppel, D. (2010) · PLoS Biology · doi:10.1371/journal.pbio.1000445 · Cited by 234
Temporal Coherence in the Perceptual Organization and Cortical Representation of Auditory Scenes
Elhilali, M., Ma, L., Micheyl, C., Oxenham, A. J., & Shamma, S. A. (2009) · Neuron · doi:10.1016/j.neuron.2008.12.005 · Cited by 228 · cited 1×
“…and theoretical perspectives on stream segregation suggest that multifeature auditory sources are segregated into perceptual objects based on the temporal coherence of the neuronal responses to the various acoustic features (Elhilali et al., 2009; Shamma et al., 2011; Ding and Simon, 2012).”
BackgroundTesting the race model inequality: An algorithm and computer programs
Ulrich, R., Miller, J., & Schröter, H. (2007) · Behavior Research Methods · doi:10.3758/bf03193160 · Cited by 203 · cited 1×
“CDFs were generated for each participant and condition, divided into nine quantiles (0.1, 0.2, . . . , 0.9) and group averaged (Ulrich et al., 2007).”
MethodInfluentialRoss, L. A., Saint-Amour, D., Leavitt, V. M., Molholm, S., Javitt, D. C., & Foxe, J. J. (2007) · Schizophrenia Research · doi:10.1016/j.schres.2007.08.008 · Cited by 186 · cited 3×
“Future work examining this using a speech-in-noise paradigm should prove more informative given the well-established benefits of multisensory speech in adverse hearing conditions (Sumby and Pollack, 1954; Ross et al., 2007a).”
BackgroundAltered temporal profile of visual–auditory multisensory interactions in dyslexia
Hairston, W. D., Burdette, J. H., Flowers, D. L., Wood, F. B., & Wallace, M. T. (2005) · Experimental Brain Research · doi:10.1007/s00221-005-2387-6 · Cited by 183
Processing of changes in visual speech in the human auditory cortex
Möttönen, R., Krause, C. M., Tiippana, K. M., & Sams, M. E. (2002) · Cognitive Brain Research · doi:10.1016/s0926-6410(02)00053-8 · Cited by 174 · cited 2×
“Traditionally, EEG/MEG studies have focused on how the brain responds to discrete AV stimuli such as syllables (Sams et al., 1991; Möttönen et al., 2002), an approach that is limited in what it can say about the role of the temporal correlation between continuous auditory and visual speech.”
BackgroundKlucharev, V., Möttönen, R., & Sams, M. (2003) · Cognitive Brain Research · doi:10.1016/j.cogbrainres.2003.09.004 · Cited by 174 · cited 2×
“Multisensory interaction effects as a function of temporal scale It has been suggested that AV speech perception includes the neuronal integration of temporally fine-grained correlations between the auditory and visual speech stimuli, even at the phonetic level (Grant and Seitz, 2000; Klucharev et al., 2003).”
BackgroundNeural pathways for visual speech perception
Bernstein, L. E., & Liebenthal, E. (2014) · Frontiers in Neuroscience · doi:10.3389/fnins.2014.00386 · Cited by 166 · cited 2×
“This would allow the AVc model to infer complementary information from correlated visual speech processing as reflected on parieto-occipital channels (Luo et al., 2010; Bernstein and Liebenthal, 2014), even without ever explicitly quantifying those visual features in the model fitting.”
BackgroundMethodMesgarani, N., David, S. V., Fritz, J. B., & Shamma, S. A. (2009) · Journal of Neurophysiology · doi:10.1152/jn.91128.2008 · Cited by 164 · cited 5×
“Indeed, the reconstruction model can also indirectly index activity in brain areas whose activity is correlated with the acoustic envelope, even if that activity is not reflected directly in the data (Mesgarani et al., 2009).”
BackgroundMethodInfluentialBesle, J., Fischer, C., Bidet-Caulet, A., Lecaignard, F., Bertrand, O., & Giard, M.-H. (2008) · The Journal of Neuroscience · doi:10.1523/jneurosci.2875-08.2008 · Cited by 144
Brain Bases for Auditory Stimulus-Driven Figure–Ground Segregation
Teki, S., Chait, M., Kumar, S., von Kriegstein, K., & Griffiths, T. D. (2011) · The Journal of Neuroscience · doi:10.1523/jneurosci.3788-10.2011 · Cited by 137 · cited 1×
“Indeed, recent research has shown evidence for neural computations in this area that underpin auditory figureground segregation using stimuli that display periods of temporal coherence across multiple frequency channels (Teki et al., 2011).”
BackgroundOn the Relationship between Face Movements, Tongue Movements, and Speech Acoustics
Jiang, J., Alwan, A. A., Keating, P., Auer, E. T., & Bernstein, L. E. (2002) · EURASIP Journal on Advances in Signal Processing · doi:10.1155/s1110865702206046 · Cited by 128 · cited 1×
“Addressing this issue is not entirely straightforward because there are many visual speech features at different levels of temporal granularity that correlate with the acoustic envelope (Jiang et al., 2002; Chandrasekaran et al., 2009; Jiang and Bernstein, 2011).”
BackgroundSpatiotemporal dynamics of audiovisual speech processing
Bernstein, L. E., Auer, E. T., Wagner, M., & Ponton, C. W. (2007) · NeuroImage · doi:10.1016/j.neuroimage.2007.08.035 · Cited by 110 · cited 2×
“4C), in line with previous studies that have demonstrated emergent multisensory interactions in the form of suppressed cortical measures at 120 –190 ms (Besle et al., 2004b) and 160 –220 ms (Bernstein et al., 2008).”
ResultDeficient multisensory integration in schizophrenia: An event-related potential study
Stekelenburg, J. J., Maes, J. P., Gool, A. R. V., Sitskoorn, M. M., & Vroomen, J. (2013) · Schizophrenia Research · doi:10.1016/j.schres.2013.04.038 · Cited by 92 · cited 1×
“This naturalistic approach may prove useful in research with clinical populations in which altered multisensory (AV) processing has been reported, e.g., dyslexia (Hairston et al., 2005), autism (Brandwein et al., 2013), and schizophrenia (Ross et al., 2007b; Stekelenburg et al., 2013).”
BackgroundKayser, C. (2009) · Frontiers in Integrative Neuroscience · doi:10.3389/neuro.07.007.2009 · Cited by 87
Besle, J., Fort, A., & Giard, M. (2004) · Cognitive Processing · doi:10.1007/s10339-004-0026-y · Cited by 87 · cited 2×
“The validity of the additive model for the purpose of indexing multisensory integration in electrophysiological studies is well established (Besle et al., 2004a).”
BackgroundResultPsychophysics of the McGurk and other audiovisual speech integration effects.
Jiang, J., & Bernstein, L. E. (2011) · Journal of Experimental Psychology Human Perception & Performance · doi:10.1037/a0023100 · Cited by 56 · cited 3×
“possible because the visible articulators that determine the vocal resonances, such as the lips, teeth, and tongue, as well as ancillary movements, such as facial, head, and hand movements, are temporally correlated with the vocalized acoustic signal (Summerfield, 1992; Grant and Seitz, 2000; Jiang and Bernstein, 2011…”
BackgroundAssessing face and speech consistency for monologue detection in video
Nock, H. J., Iyengar, G., & Neti, C. (2002) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/641007.641070 · Cited by 47 · cited 1×
“In addition, using other approaches to quantify AV correlations such as those based on mutual information models (Nock et al., 2002) and hidden Markov models (Rabiner, 1989) may provide important complementary insights.”
MethodA visual or tactile signal makes auditory speech detection more efficient by reducing uncertainty
Tjan, B. S., Chao, E., & Bernstein, L. E. (2014) · European Journal of Neuroscience · doi:10.1111/ejn.12471 · Cited by 16 · cited 1×
“In contrast, other work has suggested that, at least in some detection paradigms, neuronal integration at this detailed level of granularity is not necessary (Tjan et al., 2014).”
ResultInvestigating the temporal dynamics of auditory cortical activation to silent lipreading
Crosse, M. J., ElShafei, H. A., Foxe, J. J., & Lalor, E. C. (2015) · International IEEE/EMBS Conference on Neural Engineering · doi:10.1109/ner.2015.7146621 · Cited by 13 · cited 1×
“However, in keeping with an analysis-by-synthesis framework, Crosse et al. (2015) suggest that such occipital activity may, in fact, reflect the processing of higher-level visual speech features in visual cortex in addition to just motion tracking.”
ResultSelf-citationThe effects of attention and visual input on the representation of natural speech in EEG
O’Sullivan, J., Crosse, M. J., Power, A. J., & Lalor, E. C. (2013) · Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC) · doi:10.1109/embc.2013.6610122 · Cited by 10 · cited 1×
“Previous work has attempted to circumvent this bias by restricting the analysis to only the frontal electrodes (O’Sullivan et al., 2013).”
Self-citation
Cited by
Crosse, M. J., Di Liberto, G. M., Bednar, A., & Lalor, E. C. (2016) · Frontiers in Human Neuroscience · doi:10.3389/fnhum.2016.00604 · Cited by 705
Self-citationBroderick, M. P., Anderson, A. J., Di Liberto, G. M., Crosse, M. J., & Lalor, E. C. (2018) · Current Biology · doi:10.1016/j.cub.2018.01.080 · Cited by 421
Self-citationBlank, H., & Davis, M. H. (2016) · PLOS Biology · doi:10.1371/journal.pbio.1002577 · Cited by 181
Ronconi, L., Molteni, M., & Casartelli, L. (2016) · Frontiers in Human Neuroscience · doi:10.3389/fnhum.2016.00144 · Cited by 39
Metadata
- Authors
- Crosse, M. J., Butler, J. S., & Lalor, E. C.
- Venue
- The Journal of Neuroscience
- Publisher
- Society for Neuroscience
- Published
- 2015-10-21
- Volume
- 35
- Issue
- 42
- Pages
- 14195-14204
- ISSN
- 0270-6474 (ISSN-L)
- 1529-2401
- OpenAlex
- W2176376079
- Cited by
- 208
- References
- 67
- FWCI
- 19.79
- Type
- journal article
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.