The DOI matches its Crossref record; 67 of the 67 references it cites were found in a registry or an index.
The processing of audio-visual speech: empirical and neural bases
Open accessAbstract
In this selective review, I outline a number of ways in which seeing the talker affects auditory perception of speech, including, but not confined to, the McGurk effect. To date, studies suggest that all linguistic levels are susceptible to visual influence, and that two main modes of processing can be described: a complementary mode, whereby vision provides information more efficiently than hearing for some under-specified parts of the speech stream, and a correlated mode, whereby vision partially duplicates information about dynamic articulatory patterning. Cortical correlates of seen speech suggest that at the neurological as well as the perceptual level, auditory processing of speech is affected by vision, so that ‘auditory speech regions’ are activated by seen speech. The processing of natural speech, whether it is heard, seen or heard and seen, activates the perisylvian language regions (left>right). It is highly probable that activation occurs in a specific order. First, superior temporal, then inferior parietal and finally inferior frontal regions (left>right) are activated. There is some differentiation of the visual input stream to the core perisylvian language system, suggesting that complementary seen speech information makes special use of the visual ventral processing stream, while for correlated visual speech, the dorsal processing stream, which is sensitive to visual movement, may be relatively more involved.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Hearing lips and seeing voices
MCGURK, H., & MACDONALD, J. (1976) · Nature · doi:10.1038/264746a0 · Cited by 6,183
Fant, G. (1971) · De Gruyter eBooks · doi:10.1515/9783110873429 · Cited by 2,945
Rizzolatti, G., & Arbib, M. A. (1998) · Trends in Neurosciences · doi:10.1016/s0166-2236(98)01260-0 · Cited by 2,858
Visual Contribution to Speech Intelligibility in Noise
Sumby, W. H., & Pollack, I. (1954) · The Journal of the Acoustical Society of America · doi:10.1121/1.1907309 · Cited by 2,583 · cited 1×
“Fifty years ago, it was established that seeing the talker could give an improvement in the comprehension of auditory speech in noise equivalent to that produced by an increase of up to 15 dB in signalto-noise ratio (Sumby & Pollack 1954).”
BackgroundAction observation activates premotor and parietal areas in a somatotopic manner: an fMRI study
Buccino, G., Binkofski, F., Fink, G. R., Fadiga, L., Fogassi, L., Gallese, V., Seitz, R. J., Zilles, K., Rizzolatti, G., & Freund, H. ‐. (2001) · European Journal of Neuroscience · doi:10.1111/j.1460-9568.2001.01385.x · Cited by 2,488
Hickok, G., & Poeppel, D. (2004) · Cognition · doi:10.1016/j.cognition.2003.10.011 · Cited by 2,212
Newborn Infants Imitate Adult Facial Gestures
Meltzoff, A. N., & Moore, M. (1983) · Child Development · doi:10.1111/j.1467-8624.1983.tb00496.x · Cited by 1,307
Activation of Auditory Cortex During Silent Lipreading
Calvert, G. A., Bullmore, E. T., Brammer, M. J., Campbell, R., Williams, S. C. R., McGuire, P. K., Woodruff, P. W. R., Iversen, S. D., & David, A. S. (1997) · Science · doi:10.1126/science.276.5312.593 · Cited by 1,025
Self-citationCalvert, G. A., Campbell, R., & Brammer, M. J. (2000) · Current Biology · doi:10.1016/s0960-9822(00)00513-3 · Cited by 954
Self-citationVisual speech speeds up the neural processing of auditory speech
van Wassenhove, V., Grant, K. W., & Poeppel, D. (2005) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.0408949102 · Cited by 854
Seeing and hearing speech excites the motor system involved in speech production
Watkins, K. E., Strafella, A. P., & Paus, T. (2003) · Neuropsychologia · doi:10.1016/s0028-3932(02)00316-0 · Cited by 601
Imitation: Is cognitive neuroscience solving the correspondence problem?
Brass, M., & Heyes, C. (2005) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2005.08.007 · Cited by 580
Acoustic and Auditory Phonetics
Johnson, K., & Johnson, K. (2004) · Phonetica · doi:10.1159/000078663 · Cited by 555
Multisensory Integration of Dynamic Faces and Voices in Rhesus Monkey Auditory Cortex
Ghazanfar, A. A., Maier, J. X., Hoffman, K. L., & Logothetis, N. K. (2005) · The Journal of Neuroscience · doi:10.1523/jneurosci.0799-05.2005 · Cited by 544
Visual Prosody and Speech Intelligibility: Head Movement Improves Auditory Speech Perception
Munhall, K. G., Jones, J. A., Callan, D. E., Kuratate, T., & Vatikiotis-Bateson, E. (2004) · Psychological Science · doi:10.1111/j.0963-7214.2004.01502010.x · Cited by 482
Audiovisual Integration of Speech Falters under High Attention Demands
Alsius, A., Navarra, J., Campbell, R., & Soto-Faraco, S. (2005) · Current Biology · doi:10.1016/j.cub.2005.03.046 · Cited by 380
Self-citationRosenblum, L. D., Schmuckler, M. A., & Johnson, J. A. (1997) · Perception & Psychophysics · doi:10.3758/bf03211902 · Cited by 367
Seeing speech: visual information from lip movements modifies activity in the human auditory cortex
Sams, M., Aulanko, R., Hämäläinen, M., Hari, R., Lounasmaa, O. V., Lu, S.-T., & Simola, J. (1991) · Neuroscience Letters · doi:10.1016/0304-3940(91)90914-f · Cited by 367
Nishitani, N., & Hari, R. (2002) · Neuron · doi:10.1016/s0896-6273(02)01089-9 · Cited by 353
Reading Speech from Still and Moving Faces: The Neural Substrates of Visible Speech
Calvert, G. A., & Campbell, R. (2003) · Journal of Cognitive Neuroscience · doi:10.1162/089892903321107828 · Cited by 332
Self-citationListening to talking faces: motor cortical activation during speech perception
Skipper, J. I., Nusbaum, H. C., & Small, S. L. (2005) · NeuroImage · doi:10.1016/j.neuroimage.2004.11.006 · Cited by 324
Perceptual Fusion and Stimulus Coincidence in the Cross-Modal Integration of Speech
Miller, L. M., & D'Esposito, M. (2005) · The Journal of Neuroscience · doi:10.1523/jneurosci.0896-05.2005 · Cited by 322
Use of Visual Information for Phonetic Perception
Summerfield, Q. (1979) · Phonetica · doi:10.1159/000259969 · Cited by 319
Neural systems underlying British Sign Language and audio-visual English processing in native users
MacSweeney, M. (2002) · Brain · doi:10.1093/brain/awf153 · Cited by 317
Enhancement of selective listening by illusory mislocation of speech sounds due to lip-reading
Driver, J. (1996) · Nature · doi:10.1038/381066a0 · Cited by 315
Primary auditory cortex activation by visual speech: an fMRI study at 3???T
Pekkola, J., Ojanen, V., Autti, T., J????skel??inen, I. P., M??tt??nen, R., Tarkiainen, A., & Sams, M. (2005) · NeuroReport · doi:10.1097/00001756-200502080-00010 · Cited by 306
Polysensory Interactions along Lateral Temporal Regions Evoked by Audiovisual Speech
Wright, T. M. (2003) · Cerebral Cortex · doi:10.1093/cercor/13.10.1034 · Cited by 277
Linking facial animation, head motion and speech acoustics
Yehia, H. C., Kuratate, T., & Vatikiotis-Bateson, E. (2002) · Journal of Phonetics · doi:10.1006/jpho.2002.0165 · Cited by 270
Burnham, D., & Dodd, B. (2004) · Developmental Psychobiology · doi:10.1002/dev.20032 · Cited by 247
Easy to hear but hard to understand: A lip-reading advantage with intact auditory stimuli.
Reisberg, D., McLean, J. P., & Goldfield, A. (1987) · Cited by 246
Green, K. P., Kuhl, P. K., Meltzoff, A. N., & Stevens, E. B. (1991) · Perception & Psychophysics · doi:10.3758/bf03207536 · Cited by 239 · cited 3×
“The findings that phonetic context perceived by eye can shift phonemic category boundaries (Green et al. 1991), and that prelinguistic babies are sensitive to McGurk effects (Burnham & Dodd 2004), demonstrate that audio-visual integration can occur 'pre-phonemically'.”
BackgroundCampbell, R., MacSweeney, M., Surguladze, S., Calvert, G., McGuire, P., Suckling, J., Brammer, M. J., & David, A. S. (2001) · Cognitive Brain Research · doi:10.1016/s0926-6410(01)00054-4 · Cited by 206
Self-citationAn audiovisual test of kinematic primitives for visual speech perception.
Rosenblum, L. D., & Saldaña, H. M. (1996) · Journal of Experimental Psychology: Human Perception and Performance · doi:10.1037/0096-1523.22.2.318 · Cited by 204
Auditory speech detection in noise enhanced by lipreading
Bernstein, L. E., Auer, E. T., & Takayanagi, S. (2004) · Speech Communication · doi:10.1016/j.specom.2004.10.011 · Cited by 185
Listening with eye and hand: Cross-modal contributions to speech perception.
Fowler, C. A., & Dekle, D. J. (1991) · Journal of Experimental Psychology: Human Perception and Performance · doi:10.1037/0096-1523.17.3.816 · Cited by 183
Processing of audiovisual speech in Broca's area
Ojanen, V., Möttönen, R., Pekkola, J., Jääskeläinen, I. P., Joensuu, R., Autti, T. H., & Sams, M. E. (2005) · NeuroImage · doi:10.1016/j.neuroimage.2004.12.001 · Cited by 178
Colin, C. (2002) · Clinical Neurophysiology · doi:10.1016/s1388-2457(02)00024-x · Cited by 173
The After-Effects of Ventriloquism
Radeau, M., & Bertelson, P. (1974) · Quarterly Journal of Experimental Psychology · doi:10.1080/14640747408400388 · Cited by 171
Point-Light Facial Displays Enhance Comprehension of Speech in Noise
Rosenblum, L. D., Johnson, J. A., & Saldaña, H. M. (1996) · Journal of Speech Language and Hearing Research · doi:10.1044/jshr.3906.1159 · Cited by 156
Visual attention modulates audiovisual speech perception
Tiippana, K. M., Andersen, T., & Sams, M. E. (2004) · The European Journal of Cognitive Psychology · doi:10.1080/09541440340000268 · Cited by 154
Auditory processing — speech, space and auditory objects
Scott, S. K. (2005) · Current Opinion in Neurobiology · doi:10.1016/j.conb.2005.03.009 · Cited by 132
Seeing and hearing others and oneself talk
Sams, M. E., Möttönen, R., & Sihvonen, T. (2005) · Cognitive Brain Research · doi:10.1016/j.cogbrainres.2004.11.006 · Cited by 120
Reading Fluent Speech from Talking Faces: Typical Brain Networks and Individual Differences
Hall, D. A., Fussell, C., & Summerfield, A. Q. (2005) · Journal of Cognitive Neuroscience · doi:10.1162/0898929054021175 · Cited by 119
The human temporal lobe integrates facial form and motion: evidence from fMRI and ERP studies
Puce, A., Syngeniotis, A., Thompson, J. C., Abbott, D. F., Wheaton, K. J., & Castiello, U. (2003) · NeuroImage · doi:10.1016/s1053-8119(03)00189-7 · Cited by 118
Callan, D. E., Jones, J. A., Munhall, K. G., Kroos, C., Callan, A. M., & Vatikiotis‐Bateson, E. (2004) · Journal of Cognitive Neuroscience · doi:10.1162/089892904970771 · Cited by 115
Soto‐Faraco, S. (2004) · Cognition · doi:10.1016/j.cognition.2003.10.005 · Cited by 114
Perceiving Biological Motion: Dissociating Visible Speech from Walking
Santi, A., Servos, P., Vatikiotis‐Bateson, E., Kuratate, T., & Munhall, K. G. (2003) · Journal of Cognitive Neuroscience · doi:10.1162/089892903322370726 · Cited by 112
Visual speech perception without primary auditory cortex activation
Bernstein, L. E., Auer, E. T., Moore, J. K., Ponton, C. W., Don, M., & Singh, M. (2002) · Neuroreport · doi:10.1097/00001756-200203040-00013 · Cited by 110
Sadato, N., Okada, T., Honda, M., Matsuki, K.-I., Yoshida, M., Kashikura, K., Takei, W., Sato, T., Kochiyama, T., & Yonekura, Y. (2004) · Cerebral Cortex · doi:10.1093/cercor/bhh210 · Cited by 109
Seeing speech affects acoustic information processing in the human brainstem
Musacchia, G., Sams, M. E., Nicol, T. G., & Kraus, N. (2005) · Experimental Brain Research · doi:10.1007/s00221-005-0071-5 · Cited by 107
Pekkola, J., Laasonen, M. R., Ojanen, V., Autti, T. H., Jääskeläinen, I. P., Kujala, T., & Sams, M. E. (2005) · NeuroImage · doi:10.1016/j.neuroimage.2005.09.069 · Cited by 103
Viewing speech modulates activity in the left SI mouth cortex
Möttönen, R., Järveläinen, J., Sams, M. E., & Hari, R. (2004) · NeuroImage · doi:10.1016/j.neuroimage.2004.10.011 · Cited by 91
Möttönen, R., Schürmann, M., & Sams, M. E. (2004) · Neuroscience Letters · doi:10.1016/j.neulet.2004.03.076 · Cited by 83
Acoustic and auditory phonetics: the adaptive design of speech sound systems
Diehl, R. L. (2007) · Philosophical Transactions of the Royal Society B Biological Sciences · doi:10.1098/rstb.2007.2153 · Cited by 80
Hearing by Eye: The Psychology of Lip-Reading
Taylor, D. (1988) · British Journal of Ophthalmology · doi:10.1136/bjo.72.6.479-b · Cited by 71
Recalibration of phonetic categories by lipread speech versus lexical information.
van Linden, S., & Vroomen, J. (2007) · Journal of Experimental Psychology Human Perception & Performance · doi:10.1037/0096-1523.33.6.1483 · Cited by 71
Audiovisual Speech Binding: Convergence or Association?
Bernstein, L. E., Auer, E. T., & Moore, J. K. (2004) · The MIT Press eBooks · doi:10.7551/mitpress/3422.003.0017 · Cited by 71
Effects of Distance on Visual and Audiovisual Speech Recognition
Jordan, T. R., & Sergeant, P. (2000) · Language and Speech · doi:10.1177/00238309000430010401 · Cited by 58
Speechreading in the akinetopsic patient, L.M
Campbell, R. (1997) · Brain · doi:10.1093/brain/120.10.1793 · Cited by 55
Self-citationHemodynamic Studies of Audiovisual Interactions
Calvert, G. A., & Lewis, J. W. (2004) · The MIT Press eBooks · doi:10.7551/mitpress/3422.003.0037 · Cited by 44
The cortical organization of audio-visual sentence comprehension: an fMRI study at 4 Tesla
Capek, C. M., Bavelier, D., Corina, D. P., Newman, A. J., Jezzard, P., & Neville, H. J. (2004) · Cognitive Brain Research · doi:10.1016/j.cogbrainres.2003.10.014 · Cited by 43
Lip-Reading Ability and Patterns of Cortical Activation Studied Using fMRI
Ludman, C. N., Summerfield, A. Q., Hall, D. A., et al. (2000) · British Journal of Audiology · doi:10.3109/03005364000000132 · Cited by 41
Adaptation of neuromagnetic N1 responses to phonetic stimuli by visual speech in humans.
Jääskeläinen, I. P., Ojanen, V., Ahveninen, J., Auranen, T., Levänen, S., Möttönen, R., Tarnanen, I., & Sams, M. E. (2004) · PubMed · Cited by 41
Functional imaging of the auditory processing applied to speech sounds
Patterson, R. D., & Johnsrude, I. S. (2007) · Philosophical Transactions of the Royal Society B Biological Sciences · doi:10.1098/rstb.2007.2157 · Cited by 40
Bottom-Up Driven Speechreading in a Speechreading Expert: The Case of AA (JK023)
Andersson, U., & Lidestam, B. (2005) · Ear and Hearing · doi:10.1097/00003446-200504000-00008 · Cited by 27
Effects of Spectro-Temporal Asynchrony in Auditory and Auditory-Visual Speech Processing
Grant, K. W., Greenberg, S., Poeppel, D., & van Wassenhove, V. (2004) · Seminars in Hearing · doi:10.1055/s-2004-832858 · Cited by 23
Formant transition-specific adaptation by lipreading of left auditory cortex N1m
Jääskeläinen, I. P., Kauramäki, J., Tujunen, J., & Sams, M. E. (2008) · Neuroreport · doi:10.1097/wnr.0b013e3282f36f7a · Cited by 10
Cited by
The Ease of Language Understanding (ELU) model: theoretical, empirical, and clinical advances
Rönnberg, J., Lunner, T., Zekveld, A., Sörqvist, P., Danielsson, H., Lyxell, B., Dahlström, Ö., Signoret, C., Stenfelt, S., Pichora-Fuller, M. K., et al. (2013) · Frontiers in Systems Neuroscience · doi:10.3389/fnsys.2013.00031 · Cited by 989
The Natural Statistics of Audiovisual Speech
Chandrasekaran, C., Trubanova, A., Stillittano, S., Caplier, A., & Ghazanfar, A. A. (2009) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1000436 · Cited by 630
Integrating face and voice in person perception
Campanella, S., & Belin, P. (2007) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2007.10.001 · Cited by 527
Multimodal Integration of Carbon Dioxide and Other Sensory Cues Drives Mosquito Attraction to Humans
McMeniman, C. J., Corfas, R. A., Matthews, B. J., Ritchie, S. A., & Vosshall, L. B. (2014) · Cell · doi:10.1016/j.cell.2013.12.044 · Cited by 499
Bridging the Gap between Social Animal and Unsocial Machine: A Survey of Social Signal Processing
Vinciarelli, A., Pantic, M., Heylen, D., Pelachaud, C., Poggi, I., D'Errico, F., & Schroeder, M. (2012) · IEEE Transactions on Affective Computing · doi:10.1109/t-affc.2011.27 · Cited by 381
Metadata
- Authors
- Campbell, R.
- Venue
- Philosophical Transactions of the Royal Society B: Biological Sciences
- Publisher
- The Royal Society
- Published
- 2007-09-07
- Volume
- 363
- Issue
- 1493
- Pages
- 1001-1010
- ISSN
- 0962-8436 (ISSN-L)
- 1471-2970
- OpenAlex
- W2125081437
- Cited by
- 282
- References
- 67
- FWCI
- 13.35
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.