Neural Correlates of Multisensory Integration of Ecologically Valid Audiovisual Events
Open accessThe DOI matches its Crossref record; 54 of the 54 references it cites were found in a registry or an index.
Abstract
A question that has emerged over recent years is whether audiovisual (AV) speech perception is a special case of multi-sensory perception. Electrophysiological (ERP) studies have found that auditory neural activity (N1 component of the ERP) induced by speech is suppressed and speeded up when a speech sound is accompanied by concordant lip movements. In Experiment 1, we show that this AV interaction is not speech-specific. Ecologically valid nonspeech AV events (actions performed by an actor such as handclapping) were associated with a similar speeding-up and suppression of auditory N1 amplitude as AV speech (syllables). Experiment 2 demonstrated that these AV interactions were not influenced by whether A and V were congruent or incongruent. In Experiment 3 we show that the AV interaction on N1 was absent when there was no anticipatory visual motion, indicating that the AV interaction only occurred when visual anticipatory motion preceded the sound. These results demonstrate that the visually induced speeding-up and suppression of auditory N1 amplitude reflect multisensory integrative mechanisms of AV events that crucially depend on whether vision predicts when the sound occurs.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Rizzolatti, G., & Craighero, L. (2004) · Annual Review of Neuroscience · doi:10.1146/annurev.neuro.27.070203.144230 · Cited by 7,433 · cited 2×
“The presumed function of these mirror neurons is to mediate imitation and aid action and understanding (Rizzolatti & Craighero, 2004).”
Hearing lips and seeing voices
MCGURK, H., & MACDONALD, J. (1976) · Nature · doi:10.1038/264746a0 · Cited by 6,183 · cited 1×
“…stage, the auditory and visual streams are then combined into a multisensory representation, as can be demonstrated by the so-called McGurk illusion (McGurk & MacDonald, 1976), where listeners report to ‘‘hear’’ /da/ when, in fact, auditory /ba/ is synchronized to a face articulating /ga/.”
A new method for off-line removal of ocular artifact
Gratton, G., Coles, M. G. H., & Donchin, E. (1983) · Electroencephalography and Clinical Neurophysiology · doi:10.1016/0013-4694(83)90135-9 · Cited by 5,055 · cited 1×
“After electrooculogram correction (Gratton, Coles, & Donchin, 1983), epochs with an amplitude change exceeding ±150 A V at any channel were rejected.”
Cattaneo, L., & Rizzolatti, G. (2009) · Archives of Neurology · doi:10.1001/archneurol.2009.41 · Cited by 3,137
Cortical mechanisms of human imitation
Iacoboni, M., Woods, R. P., Brass, M., Bekkering, H., Mazziotta, J. C., & Rizzolatti, G. (1999) · Science · doi:10.1126/science.286.5449.2526 · Cited by 2,683 · cited 1×
“Interestingly, Broca’s area is not only active during AV speech but is also responsive to perception and imitation of meaningful goal-directed hand movements (Koski et al., 2002; Grezes, Costes, & Decety, 1999; Iacoboni et al., 1999).”
Risto, N. (2018) · Routledge eBooks · doi:10.4324/9780429487354 · Cited by 2,024
McCarthy, G., & Wood, C. C. (1985) · Electroencephalography and Clinical Neurophysiology/Evoked Potentials Section · doi:10.1016/0168-5597(85)90015-2 · Cited by 1,756
Giard, M. H., & Peronnet, F. (1999) · Journal of Cognitive Neuroscience · doi:10.1162/089892999563544 · Cited by 1,072 · cited 4×
“…that the suppression of auditory N1 is exclusively related to the integration of AV speech, because this was not found in simplified AV combinations such as pure tones and geometrical shapes (Fort et al., 2002; Giard & Peronnet, 1999), or spoken and written forms (Raij, Uutela, & Hari, 2000).”
InfluentialCalvert, G. A., Campbell, R., & Brammer, M. J. (2000) · Current Biology · doi:10.1016/s0960-9822(00)00513-3 · Cited by 954 · cited 1×
“Hemodynamic studies have shown that multisensory cortices (superior temporal sulcus/gyrus) (Skipper, Nusbaum, & Small, 2005; Callan et al., 2004; Calvert, Campbell, & Brammer, 2000) and ‘‘sensory-specific’’ cortices (Von Kriegstein & Giraud, 2006; Gonzalo & Bu¨chel, 2004; Callan et al., 2003;…”
Molholm, S., Ritter, W., Murray, M. M., Javitt, D. C., Schroeder, C. E., & Foxe, J. J. (2002) · Cognitive Brain Research · doi:10.1016/s0926-6410(02)00066-6 · Cited by 902
Adler, L. E., Pachtman, E., Franks, R. D., Pecevich, M., Waldo, M. C., & Freedman, R. (1982) · PubMed · Cited by 896
Listening to speech activates motor areas involved in speech production
Wilson, S. M., Saygin, A. P., Sereno, M. I., & Iacoboni, M. (2004) · Nature Neuroscience · doi:10.1038/nn1263 · Cited by 885 · cited 1×
“Broca’s area is not only involved in speech production (Heim, Opitz, Muller, & Friederici, 2003) but is also activated during silent lipreading (Campbell et al., 2001) and passive listening of auditory speech ( Wilson, Saygin, Sereno, & Iacoboni, 2004 ).”
Visual speech speeds up the neural processing of auditory speech
van Wassenhove, V., Grant, K. W., & Poeppel, D. (2005) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.0408949102 · Cited by 854
(1983) · Neuropsychobiology · doi:10.1159/000117925 · Cited by 840
Integration of Auditory and Visual Information about Objects in Superior Temporal Sulcus
Beauchamp, M. S., Lee, K. E., Argall, B. D., & Martin, A. (2004) · Neuron · doi:10.1016/s0896-6273(04)00070-4 · Cited by 773
Significance Testing of Difference Potentials
Guthrie, D., & Buchwald, J. S. (1991) · Psychophysiology · doi:10.1111/j.1469-8986.1991.tb00417.x · Cited by 748
Perceiving talking faces: from speech perception to a behavioral principle
Massaro, D. W. (1997) · Medical Entomology and Zoology · Cited by 615
Multisensory Visual-Auditory Object Recognition in Humans: a High-density Electrical Mapping Study
Molholm, S. (2004) · Cerebral Cortex · doi:10.1093/cercor/bhh007 · Cited by 384
Seeing speech: visual information from lip movements modifies activity in the human auditory cortex
Sams, M., Aulanko, R., Hämäläinen, M., Hari, R., Lounasmaa, O. V., Lu, S.-T., & Simola, J. (1991) · Neuroscience Letters · doi:10.1016/0304-3940(91)90914-f · Cited by 367
Reading Speech from Still and Moving Faces: The Neural Substrates of Visible Speech
Calvert, G. A., & Campbell, R. (2003) · Journal of Cognitive Neuroscience · doi:10.1162/089892903321107828 · Cited by 332
Response amplification in sensory-specific cortices during crossmodal binding
Calvert, G. A., Brammer, M. J., Bullmore, E. T., Campbell, R., Iversen, S. D., & David, A. S. (1999) · NeuroReport · doi:10.1097/00001756-199908200-00033 · Cited by 331
Teder-Sälejärvi, W. A., McDonald, J. J., Di Russo, F., & Hillyard, S. A. (2002) · Cognitive Brain Research · doi:10.1016/s0926-6410(02)00065-4 · Cited by 330
Listening to talking faces: motor cortical activation during speech perception
Skipper, J. I., Nusbaum, H. C., & Small, S. L. (2005) · NeuroImage · doi:10.1016/j.neuroimage.2004.11.006 · Cited by 324
Bimodal speech: early suppressive visual effects in human auditory cortex
Besle, J., Fort, A., Delpuech, C., & Giard, M. (2004) · European Journal of Neuroscience · doi:10.1111/j.1460-9568.2004.03670.x · Cited by 323
Modulation of Motor and Premotor Activity during Imitation of Target-directed Actions
Koski, L. (2002) · Cerebral Cortex · doi:10.1093/cercor/12.8.847 · Cited by 316
Implicit multisensory associations influence voice recognition
von Kriegstein, K., & Giraud, A.-L. (2006) · PLoS Biology · doi:10.1371/journal.pbio.0040326 · Cited by 309 · cited 2×
“Hemodynamic studies have shown that multisensory cortices (superior temporal sulcus/gyrus) (Skipper, Nusbaum, & Small, 2005; Callan et al., 2004; Calvert, Campbell, & Brammer, 2000) and ‘‘sensory-specific’’ cortices ( Von Kriegstein & Giraud, 2006; Gonzalo & Bu ¨chel, 2004; Callan et al., 2003; Calvert et al., 1999) a…”
Audiovisual Integration of Letters in the Human Brain
Raij, T., Uutela, K., & Hari, R. (2000) · Neuron · doi:10.1016/s0896-6273(00)00138-0 · Cited by 282
Sensory gating in schizophrenics and normal controls: Effects of changing stimulation interval
Nagamoto, H. T., Adler, L. E., Waldo, M. C., & Freedman, R. (1989) · Biological Psychiatry · doi:10.1016/0006-3223(89)90215-1 · Cited by 257
Semantics and the multisensory brain: How meaning modulates processes of audio-visual integration
Doehrmann, O., & Naumer, M. J. (2008) · Brain Research · doi:10.1016/j.brainres.2008.03.071 · Cited by 228
Grèzes, J., Costes, N., & Decety, J. (1999) · Brain · doi:10.1093/brain/122.10.1875 · Cited by 210
Campbell, R., MacSweeney, M., Surguladze, S., Calvert, G., McGuire, P., Suckling, J., Brammer, M. J., & David, A. S. (2001) · Cognitive Brain Research · doi:10.1016/s0926-6410(01)00054-4 · Cited by 206
Neural processes underlying perceptual enhancement by visual speech gestures
Callan, D. E., Jones, J. A., Munhall, K., Callan, A. M., Kroos, C., & Vatikiotis-Bateson, E. (2003) · NeuroReport · doi:10.1097/00001756-200312020-00016 · Cited by 190
Processing of audiovisual speech in Broca's area
Ojanen, V., Möttönen, R., Pekkola, J., Jääskeläinen, I. P., Joensuu, R., Autti, T. H., & Sams, M. E. (2005) · NeuroImage · doi:10.1016/j.neuroimage.2004.12.001 · Cited by 180
Early auditory–visual interactions in human cortex during nonredundant target identification
Fort, A., Delpuech, C., Pernier, J., & Giard, M.-H. (2002) · Cognitive Brain Research · doi:10.1016/s0926-6410(02)00058-7 · Cited by 176
Processing of changes in visual speech in the human auditory cortex
Möttönen, R., Krause, C. M., Tiippana, K. M., & Sams, M. E. (2002) · Cognitive Brain Research · doi:10.1016/s0926-6410(02)00053-8 · Cited by 174
Klucharev, V., Möttönen, R., & Sams, M. (2003) · Cognitive Brain Research · doi:10.1016/j.cogbrainres.2003.09.004 · Cited by 174
Colin, C. (2002) · Clinical Neurophysiology · doi:10.1016/s1388-2457(02)00024-x · Cited by 173
Heim, S., Opitz, B., Müller, K., & Friederici, A. D. (2003) · Cognitive Brain Research · doi:10.1016/s0926-6410(02)00284-7 · Cited by 157
Audio–visual speech perception is special
Tuomainen, J., Andersen, T., Tiippana, K. M., & Sams, M. E. (2004) · Cognition · doi:10.1016/j.cognition.2004.10.004 · Cited by 146
Barth, D. S., Goldberg, N., Brett, B. A., & Di, S. (1995) · Brain Research · doi:10.1016/0006-8993(95)00182-p · Cited by 137
Seeing speech affects acoustic information processing in the human brainstem
Musacchia, G., Sams, M. E., Nicol, T. G., & Kraus, N. (2005) · Experimental Brain Research · doi:10.1007/s00221-005-0071-5 · Cited by 120
Callan, D. E., Jones, J. A., Munhall, K., Kroos, C., Callan, A. M., & Vatikiotis-Bateson, E. (2004) · Journal of Cognitive Neuroscience · doi:10.1162/089892904970771 · Cited by 115
Johannesen, J., Kieffaber, P. D., O’Donnell, B. F., Shekhar, A., Evans, J. D., & Hetrick, W. P. (2005) · Schizophrenia Research · doi:10.1016/j.schres.2005.05.022 · Cited by 102
Möttönen, R., Schürmann, M., & Sams, M. E. (2004) · Neuroscience Letters · doi:10.1016/j.neulet.2004.03.076 · Cited by 91
Magnée, M. J. C. M., De Gelder, B., Van Engeland, H., & Kemner, C. (2008) · Journal of Child Psychology and Psychiatry · doi:10.1111/j.1469-7610.2008.01902.x · Cited by 81
Lebib, R., Papo, D., de Bode, S., & Baudonnière, P.-M. (2003) · Neuroscience Letters · doi:10.1016/s0304-3940(03)00131-9 · Cited by 59
Arnfred, S., Eder, D. N., Hemmingsen, R., Glenthoj, B. Y., & Chen, A. C. N. (2001) · Psychiatry Research · doi:10.1016/s0165-1781(01)00226-8 · Cited by 54
Adaptation of neuromagnetic N1 responses to phonetic stimuli by visual speech in humans.
Jääskeläinen, I. P., Ojanen, V., Ahveninen, J., Auranen, T., Levänen, S., Möttönen, R., Tarnanen, I., & Sams, M. E. (2004) · PubMed · Cited by 46
Suppression of P50 evoked potential component, schizotypal beliefs and smoking
Croft, R. J., Dimoska, A., Gonsalvez, C. J., & Clarke, A. R. (2004) · Psychiatry Research · doi:10.1016/j.psychres.2004.05.009 · Cited by 41
Modification of sudden onset auditory ERP by involuntary attention to visual stimuli
Oray, S., Lu, Z., & Dawson, M. E. (2002) · International Journal of Psychophysiology · doi:10.1016/s0167-8760(01)00174-x · Cited by 36
Davis, C., Kislyuk, D. S., Kim, J., & Sams, M. E. (2008) · Brain Research · doi:10.1016/j.brainres.2008.04.077 · Cited by 29
Miki, K., Watanabe, S., & Kakigi, R. (2004) · Neuroscience Letters · doi:10.1016/j.neulet.2003.12.082 · Cited by 26
Reduced P50 auditory sensory gating response in professional musicians
Kızkın, S., Karlidağ, R., Özcan, C., & Özışık, H. İ. (2006) · Brain and Cognition · doi:10.1016/j.bandc.2006.01.006 · Cited by 22
A mixed modality paradigm for recording somatosensory and auditory P50 gating
Arnfred, S., Chen, A. C. N., Eder, D. N., Glenthoj, B. Y., & Hemmingsen, R. (2001) · Psychiatry Research · doi:10.1016/s0165-1781(01)00316-x · Cited by 16
Cited by
Cortical oscillations and sensory predictions
Arnal, L. H., & Giraud, A.-L. (2012) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2012.05.003 · Cited by 1,114
Perception of intersensory synchrony: A tutorial review
Vroomen, J., & Keetels, M. (2010) · Attention, Perception, & Psychophysics · doi:10.3758/app.72.4.871 · Cited by 464
Self-citationSpence, C., & Parise, C. (2010) · Consciousness and Cognition · doi:10.1016/j.concog.2009.12.001 · Cited by 267
Audiovisual synchrony and temporal order judgments: Effects of experimental method and stimulus type
van Eijk, R. L. J., Kohlrausch, A., Juola, J. F., & van de Par, S. (2008) · Perception & Psychophysics · doi:10.3758/pp.70.6.955 · Cited by 261
Dual Neural Routing of Visual Facilitation in Speech Processing
Arnal, L. H., Morillon, B., Kell, C. A., & Giraud, A.-L. (2009) · The Journal of Neuroscience · doi:10.1523/jneurosci.3194-09.2009 · Cited by 241
Metadata
- Authors
- Stekelenburg, J. J., & Vroomen, J.
- Venue
- Journal of Cognitive Neuroscience
- Publisher
- MIT Press - Journals
- Published
- 2007-12-01
- Volume
- 19
- Issue
- 12
- Pages
- 1964-1973
- ISSN
- 0898-929X (ISSN-L)
- 1530-8898
- OpenAlex
- W2157187516
- Cited by
- 336
- References
- 54
- FWCI
- 18.35
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.