The DOI matches its Crossref record; 84 of the 84 references it cites were found in a registry or an index.
Hearing Lips and Seeing Voices: How Cortical Areas Supporting Speech Production Mediate Audiovisual Speech Perception
Open accessAbstract
Observing a speaker's mouth profoundly influences speech perception. For example, listeners perceive an "illusory" "ta" when the video of a face producing /ka/ is dubbed onto an audio /pa/. Here, we show how cortical areas supporting speech production mediate this illusory percept and audiovisual (AV) speech perception more generally. Specifically, cortical activity during AV speech perception occurs in many of the same areas that are active during speech production. We find that different perceptions of the same syllable and the perception of different syllables are associated with different distributions of activity in frontal motor areas involved in speech production. Activity patterns in these frontal motor areas resulting from the illusory "ta" percept are more similar to the activity patterns evoked by AV(/ta/) than they are to patterns evoked by AV(/pa/) or AV(/ka/). In contrast to the activity in frontal motor areas, stimulus-evoked activity for the illusory "ta" in auditory and somatosensory areas and visual areas initially resembles activity evoked by AV(/pa/) and AV(/ka/), respectively. Ultimately, though, activity in these regions comes to resemble activity evoked by AV(/ta/). Together, these results suggest that AV speech elicits in the listener a motor plan for the production of the phoneme that the speaker might have been attempting to produce, and that feedback in the form of efference copy from the motor system ultimately influences the phonetic interpretation.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The assessment and analysis of handedness: The Edinburgh inventory
Oldfield, R. C. (1971) · Neuropsychologia · doi:10.1016/0028-3932(71)90067-4 · Cited by 35,900
AFNI: Software for analysis and visualization of functional magnetic resonance neuroimages
Cox, R. W. (1996) · Computers and Biomedical Research · doi:10.1006/cbmr.1996.0014 · Cited by 10,913
Cortical Surface-Based Analysis
Dale, A. M., Fischl, B., & Sereno, M. I. (1999) · NeuroImage · doi:10.1006/nimg.1998.0395 · Cited by 10,910
Rizzolatti, G., & Craighero, L. (2004) · Annual Review of Neuroscience · doi:10.1146/annurev.neuro.27.070203.144230 · Cited by 7,433
Hearing lips and seeing voices
MCGURK, H., & MACDONALD, J. (1976) · Nature · doi:10.1038/264746a0 · Cited by 6,183
Cortical Surface-Based Analysis
Fischl, B., Sereno, M. I., & Dale, A. M. (1999) · NeuroImage · doi:10.1006/nimg.1998.0396 · Cited by 6,147
Thresholding of statistical maps in functional neuroimaging using the false discovery rate
Genovese, C. R., Lazar, N. A., & Nichols, T. (2002) · NeuroImage · doi:10.1006/nimg.2001.1037 · Cited by 4,923
Receptive fields of single neurones in the cat's striate cortex
Hubel, D. H., & Wiesel, T. N. (1959) · The Journal of Physiology · doi:10.1113/jphysiol.1959.sp006308 · Cited by 4,653
Automatically Parcellating the Human Cerebral Cortex
Fischl, B. (2004) · Cerebral Cortex · doi:10.1093/cercor/bhg087 · Cited by 4,253
Distributed and overlapping representations of faces and objects in ventral temporal cortex
Haxby, J. V., Gobbini, M. I., Furey, M. L., Ishai, A., Schouten, J. L., & Pietrini, P. (2001) · Science · doi:10.1126/science.1063736 · Cited by 3,940
Perception of the speech code.
Liberman, A. M., Cooper, F. S., Shankweiler, D. P., & Studdert-Kennedy, M. (1967) · Psychological Review · doi:10.1037/h0020279 · Cited by 3,722
Cattaneo, L., & Rizzolatti, G. (2009) · Archives of Neurology · doi:10.1001/archneurol.2009.41 · Cited by 3,137
The TRACE model of speech perception
McClelland, J. L., & Elman, J. L. (1986) · Cognitive Psychology · doi:10.1016/0010-0285(86)90015-0 · Cited by 2,956
The motor theory of speech perception revised
Liberman, A. M., & Mattingly, I. G. (1985) · Cognition · doi:10.1016/0010-0277(85)90021-6 · Cited by 2,861
Das Reafferenzprinzip: Wechselwirkungen zwischen Zentralnervensystem und Peripherie
von Holst, E., & Mittelstaedt, H. (1950) · Naturwissenschaften · doi:10.1007/bf00622503 · Cited by 2,655 · cited 1×
“Activated motor commands then predict the acoustic and somatosensory consequences of executing a speech movement through efference copy (von Holst and Mittelstaedt 1950; sometimes referred to as ''corollary discharge ,'' Sperry 1950 or ''forward models'' in the motor control literature, Jordan and…”
BackgroundFerrier lecture. Functional architecture of macaque monkey visual cortex
Hubel, D. H., & Wiesel, T. N. (1977) · Proceedings of the Royal Society of London. Series B. Biological Sciences · doi:10.1098/rspb.1977.0085 · Cited by 2,608 · cited 2×
“Blumstein 1978; Klatt 1979; Liberman and Mattingly 1985; McClelland and Elman 1986) and indeed theories of perception more broadly ( Hubel and Wiesel 1959, 1977; Barlow 1972; Barlow et al. 1972) argue for a direct mapping of stimulus signals (e.g., acoustic properties) onto perception (e.g., phonetic categories).”
BackgroundVisual Contribution to Speech Intelligibility in Noise
Sumby, W. H., & Pollack, I. (1954) · The Journal of the Acoustical Society of America · doi:10.1121/1.1907309 · Cited by 2,583 · cited 1×
“Adding visible facial movements to speech enhances speech recognition comparable with removing up to 20 dB of noise from the auditory signal (Sumby and Pollack 1954).”
BackgroundHandbuch der physiologischen Optik
Helmholtz, H. L. F. (2011) · DFG Viewer (DFG Viewer) · Cited by 1,947
Neural basis of the spontaneous optokinetic response produced by visual inversion
Sperry, R. W. (1950) · Journal of Comparative and Physiological Psychology · doi:10.1037/h0055479 · Cited by 1,789
Voice-selective areas in human auditory cortex
Belin, P., Zatorre, R. J., Lafaille, P., Ahad, P., & Pike, B. (2000) · Nature · doi:10.1038/35002078 · Cited by 1,771
Single Units and Sensation: A Neuron Doctrine for Perceptual Psychology?
Barlow, H. B. (1972) · Perception · doi:10.1068/p010371 · Cited by 1,730
Central cancellation of self-produced tickle sensation
Blakemore, S.-J., Wolpert, D. M., & Frith, C. D. (1998) · Nature Neuroscience · doi:10.1038/2870 · Cited by 1,467
Forward models: supervised learning with a distal teacher
Jordan, M. I., & Rumelhart, D. E. (1992) · Cognitive Science · doi:10.1207/s15516709cog1603_1 · Cited by 1,451
Deconvolution of Impulse Response in Event-Related BOLD fMRI1
Glover, G. H. (1999) · NeuroImage · doi:10.1006/nimg.1998.0419 · Cited by 1,386
Spectral and Temporal Processing in Human Auditory Cortex
Zatorre, R. J. (2001) · Cerebral Cortex · doi:10.1093/cercor/11.10.946 · Cited by 1,278
Wolpert, D. M., & Flanagan, J. R. (2001) · Current Biology · doi:10.1016/s0960-9822(01)00432-8 · Cited by 1,257
Models for the perception of speech and visual form
Brubaker, R. S. (1968) · Journal of Communication Disorders · doi:10.1016/0021-9924(68)90015-4 · Cited by 1,254
Listening to speech activates motor areas involved in speech production
Wilson, S. M., Saygin, A. P., Sereno, M. I., & Iacoboni, M. (2004) · Nature Neuroscience · doi:10.1038/nn1263 · Cited by 885
Talker Variability in Speech Processing
Johnson, K. A., & Mullennix, J. W. (1997) · Academic Press eBooks · Cited by 854
Visual speech speeds up the neural processing of auditory speech
van Wassenhove, V., Grant, K. W., & Poeppel, D. (2005) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.0408949102 · Cited by 854
Self-citationNeural Circuits Involved in the Recognition of Actions Performed by Nonconspecifics: An fMRI Study
Buccino, G., Lui, F., Canessa, N., Patteri, I., Lagravinese, G., Benuzzi, F., Porro, C. A., & Rizzolatti, G. (2004) · Journal of Cognitive Neuroscience · doi:10.1162/089892904322755601 · Cited by 762
Maintaining internal representations: the role of the human superior parietal lobe
Wolpert, D. M., Goodbody, S. J., & Husain, M. (1998) · Nature Neuroscience · doi:10.1038/2245 · Cited by 745
Motor cortex maps articulatory features of speech sounds
Pulvermüller, F., Huss, M., Kherif, F., Moscoso del Prado Martin, F., Hauk, O., & Shtyrov, Y. (2006) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.0509989103 · Cited by 658
Seeing and hearing speech excites the motor system involved in speech production
Watkins, K. E., Strafella, A. P., & Paus, T. (2003) · Neuropsychologia · doi:10.1016/s0028-3932(02)00316-0 · Cited by 601
Invariant cues for place of articulation in stop consonants
Stevens, K. N., & Blumstein, S. E. (1978) · The Journal of the Acoustical Society of America · doi:10.1121/1.382102 · Cited by 561
Speech perception: a model of acoustic–phonetic analysis and lexical access
Klatt, D. H. (1979) · Journal of Phonetics · doi:10.1016/s0095-4470(19)31059-9 · Cited by 486
Reafferent copies of imitated actions in the right superior temporal cortex
Iacoboni, M., Koski, L. M., Brass, M., Bekkering, H., Woods, R. P., Dubeau, M.-C., Mazziotta, J. C., & Rizzolatti, G. (2001) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.241474598 · Cited by 466
Human temporal-lobe response to vocal sounds
Belin, P., Zatorre, R. J., & Ahad, P. (2002) · Cognitive Brain Research · doi:10.1016/s0926-6410(01)00084-2 · Cited by 452 · cited 1×
“Auditory speech alone evoked far less activity in the motor system than AV speech, which is typical in speech perception studies that involve no explicit motor responses on the part of the listener (compare Zatorre et al. 1996; Belin et al. 2000, 2002; Burton et al. 2000; Zatorre and Belin 2001).”
BackgroundPET Studies of Phonetic Processing of Speech: Review, Replication, and Reanalysis
Zatorre, R. J., Meyer, E., Gjedde, A., & Evans, A. C. (1996) · Cerebral Cortex · doi:10.1093/cercor/6.1.21 · Cited by 422
Predicting the Effects of Actions: Interactions of Perception and Action
Knoblich, G., & Flach, R. (2001) · Psychological Science · doi:10.1111/1467-9280.00387 · Cited by 405
The role of segmentation in phonological processing: an fMRI investigation
Burton, M. W., Small, S. L., & Blumstein, S. E. (2000) · Journal of Cognitive Neuroscience · doi:10.1162/089892900562309 · Cited by 378
Self-citationNishitani, N., & Hari, R. (2002) · Neuron · doi:10.1016/s0896-6273(02)01089-9 · Cited by 353
RESPONSES OF AUDITORY CORTICAL NEURONS TO STIMULI OF CHANGING FREQUENCY
Whitfield, I. C., & Evans, E. F. (1965) · Journal of Neurophysiology · doi:10.1152/jn.1965.28.4.655 · Cited by 341
Reading Speech from Still and Moving Faces: The Neural Substrates of Visible Speech
Calvert, G. A., & Campbell, R. (2003) · Journal of Cognitive Neuroscience · doi:10.1162/089892903321107828 · Cited by 332
Cortical thickness analysis in autism with heat kernel smoothing
Chung, M. K., Robbins, S. M., Dalton, K. M., Davidson, R. J., Alexander, A. L., & Evans, A. C. (2005) · NeuroImage · doi:10.1016/j.neuroimage.2004.12.052 · Cited by 326
Listening to talking faces: motor cortical activation during speech perception
Skipper, J. I., Nusbaum, H. C., & Small, S. L. (2005) · NeuroImage · doi:10.1016/j.neuroimage.2004.11.006 · Cited by 324
Self-citationConnecting mirror neurons and forward models
Miall, R. C. (2003) · NeuroReport · doi:10.1097/00001756-200312020-00001 · Cited by 299
Caviness, V. S., Meyer, J., Makris, N., & Kennedy, D. N. (1996) · Journal of Cognitive Neuroscience · doi:10.1162/jocn.1996.8.6.566 · Cited by 297
Modulation of Motor Excitability during Speech Perception: The Role of Broca's Area
Watkins, K., & Paus, T. (2004) · Journal of Cognitive Neuroscience · doi:10.1162/0898929041502616 · Cited by 266
Spiral K‐space MR imaging of cortical activation
Noll, D. C., Cohen, J. D., Meyer, C. H., & Schneider, W. (1995) · Journal of Magnetic Resonance Imaging · doi:10.1002/jmri.1880050112 · Cited by 254
Easy to hear but hard to understand: A lip-reading advantage with intact auditory stimuli.
Reisberg, D., McLean, J. P., & Goldfield, A. (1987) · Cited by 246
An Efference Copy Which Is Modified by Reafferent Input
Bell, C. C. (1981) · Science · doi:10.1126/science.7291985 · Cited by 244
Callan, D. E., Jones, J. A., Callan, A. M., & Akahane-Yamada, R. (2004) · NeuroImage · doi:10.1016/j.neuroimage.2004.03.006 · Cited by 236
Action to Language via the Mirror Neuron System
(2006) · Cambridge University Press eBooks · doi:10.1017/cbo9780511541599 · Cited by 229
Wilson, S. M., & Iacoboni, M. (2006) · NeuroImage · doi:10.1016/j.neuroimage.2006.05.032 · Cited by 228
Campbell, R., MacSweeney, M., Surguladze, S., Calvert, G., McGuire, P., Suckling, J., Brammer, M. J., & David, A. S. (2001) · Cognitive Brain Research · doi:10.1016/s0926-6410(01)00054-4 · Cited by 206
The determination of molecular orbitals
Lennard-Jones, J. E. (1949) · Proceedings of the Royal Society of London. Series A. Mathematical and Physical Sciences · doi:10.1098/rspa.1949.0083 · Cited by 206
SUMA: An interface for surface-based intra- and inter-subject analysis with AFNI
Saad, Z. S., Reynolds, R. C., Argall, B., Japee, S., & Cox, R. W. (2005) · 2004 2nd IEEE International Symposium on Biomedical Imaging: Macro to Nano (IEEE Cat No. 04EX821) · doi:10.1109/isbi.2004.1398837 · Cited by 204
Corticofugal Modulation of Time-Domain Processing of Biosonar Information in Bats
Yan, J., & Suga, N. (1996) · Science · doi:10.1126/science.273.5278.1100 · Cited by 196
Neural Attenuation of Responses to Emitted Sounds in Echolocating Bats
Suga, N., & Schlegel, P. (1972) · Science · doi:10.1126/science.177.4043.82 · Cited by 193
Neural processes underlying perceptual enhancement by visual speech gestures
Callan, D. E., Jones, J. A., Munhall, K., Callan, A. M., Kroos, C., & Vatikiotis-Bateson, E. (2003) · NeuroReport · doi:10.1097/00001756-200312020-00016 · Cited by 190
Callan, D. E., Tajima, K., Callan, A. M., Kubo, R., Masaki, S., & Akahane-Yamada, R. (2003) · NeuroImage · doi:10.1016/s1053-8119(03)00020-x · Cited by 183
A comparison of bound and unbound audio–visual information processing in the human cerebral cortex
Olson, I. R., Gatenby, J. C., & Gore, J. C. (2002) · Cognitive Brain Research · doi:10.1016/s0926-6410(02)00067-8 · Cited by 160
Seeing and hearing others and oneself talk
Sams, M. E., Möttönen, R., & Sihvonen, T. (2005) · Cognitive Brain Research · doi:10.1016/j.cogbrainres.2004.11.006 · Cited by 133
A Functional-Anatomical Model for Lipreading
Paulesu, E., Perani, D., Blasi, V., Silani, G., Borghese, N. A., Giovanni, U. D., Sensolo, S., & Fazio, F. (2003) · Journal of Neurophysiology · doi:10.1152/jn.00926.2002 · Cited by 133
An intracellular study of muscle primary afferents during fictive locomotion in the cat
Gossard, J., Cabelguen, J., & Rossignol, S. (1991) · Journal of Neurophysiology · doi:10.1152/jn.1991.65.4.914 · Cited by 130
Perception as Unconscious Inference
Hatfield, G. (2002) · Wiley eBooks · doi:10.1002/0470013427.ch5 · Cited by 122
Sillar, K. T., & Skorupski, P. (1986) · Journal of Neurophysiology · doi:10.1152/jn.1986.55.4.678 · Cited by 122
A neuronal mechanism for sensory gating during locomotion in a vertebrate
Sillar, K. T., & Roberts, A. (1988) · Nature · doi:10.1038/331262a0 · Cited by 121 · cited 2×
“There is considerable behavioral (Knoblich and Flach 2001; Wolpert and Flanagan 2001) and neurophysiological (Zaretsky and Rowell 1979; Bell 1981; Sillar and Skorupski 1986; Sillar and Roberts 1988; Gossard et al. 1991; Wolf and Burrows 1995; el Manira et al. 1996; Blakemore et al. 1998; Wolpert et al. 1998; Bodznick…”
BackgroundMotor activation from visible speech: Evidence from stimulus response compatibility.
Kerzel, D., & Harold Bekkering, A. (2000) · Journal of Experimental Psychology Human Perception & Performance · doi:10.1037//0096-1523.26.2.634 · Cited by 114
Site of Neural Attenuation of Responses to Self-Vocalized Sounds in Echolocating Bats
Suga, N., & Shimozawa, T. (1974) · Science · doi:10.1126/science.183.4130.1211 · Cited by 110
Observation???execution matching system for speech: a magnetic stimulation study
Sundara, M., Namasivayam, A. K., & Chen, R. E. (2001) · Neuroreport · doi:10.1097/00001756-200105250-00010 · Cited by 104
Pekkola, J., Laasonen, M. R., Ojanen, V., Autti, T. H., Jääskeläinen, I. P., Kujala, T., & Sams, M. E. (2005) · NeuroImage · doi:10.1016/j.neuroimage.2005.09.069 · Cited by 103
Visual Pattern Analysis in Machines and Animals
Barlow, H. B., Narasimhan, R., & Rosenfeld, A. (1972) · Science · doi:10.1126/science.177.4049.567 · Cited by 100
Adaptive mechanisms in the elasmobranch hindbrain
Bodznick, D., Montgomery, J. C., & Carey, M. (1999) · Journal of Experimental Biology · doi:10.1242/jeb.202.10.1357 · Cited by 94
SPEECH INTELLIGIBILITY DERIVED FROM ASYNCHRONOUS PROCESSING OF AUDITORY-VISUAL INFORMATION
Grant, K. W., & Greenberg, S. (2001) · AVSP · Cited by 90
Phonetic features and acoustic invariance in speech
Blumstein, S. E., & Stevens, K. N. (1981) · Cognition · doi:10.1016/0010-0277(81)90021-4 · Cited by 87
Sumby, W. H., & Pollack, I. W. (1954) · The Journal of the Acoustical Society of America · doi:10.1121/1.1907384 · Cited by 76
Wolf, H., & Burrows, M. (1995) · Journal of Neuroscience · doi:10.1523/jneurosci.15-08-05623.1995 · Cited by 74
Neurophysiological Feature Detectors and Speech Perception: A Discussion of Theoretical Implications
Abbs, J. H., & Sussman, H. M. (1971) · Journal of Speech and Hearing Research · doi:10.1044/jshr.1401.23 · Cited by 72
Lending a helping hand to hearing: another motor theory of speech perception
Skipper, J. I., Nusbaum, H. C., & Small, S. L. (2006) · Cambridge University Press eBooks · doi:10.1017/cbo9780511541599.009 · Cited by 70
Self-citationAutomatic audiovisual integration in speech perception
Gentilucci, M., & Cattaneo, L. (2005) · Experimental Brain Research · doi:10.1007/s00221-005-0008-z · Cited by 69
Saccadic suppression by corollary discharge in the locust
ZARETSKY, M., & ROWELL, C. H. F. (1979) · Nature · doi:10.1038/280583a0 · Cited by 58 · cited 1×
“…center of mass of significantly greater cortical activity for the AV syllables (i.e., /pa/, /ka/, or/ta/) compared with the McGurk--MacDonald syllable (A p V k ) for the group Because A p V k is not a naturally spoken syllable and could result in a different pattern of activity compared with the…”
ResultHigh-resolution intersubject averaging and a coordinate system for the cortical surface
Fischl, B., Sereno, M. I., Tootell, R. B. H., & Dale, A. M. (1999) · Human Brain Mapping · doi:10.1002/(sici)1097-0193(1999)8:4<272::aid-hbm10>3.0.co;2-4
Cited by
An integrated theory of language production and comprehension
Pickering, M. J., & Garrod, S. (2013) · Behavioral and Brain Sciences · doi:10.1017/s0140525x12001495 · Cited by 1,569
Predictive top-down integration of prior knowledge during speech perception
Sohoglu, E., Peelle, J. E., Carlyon, R. P., & Davis, M. H. (2012) · Journal of Neuroscience · doi:10.1523/jneurosci.5069-11.2012 · Cited by 388
The Representation of Tool Use in Humans and Monkeys: Common and Uniquely Human Features
Peeters, R., Simone, L., Nelissen, K., Fabbri-Destro, M., Vanduffel, W., Rizzolatti, G., & Orban, G. A. (2009) · The Journal of Neuroscience · doi:10.1523/jneurosci.2040-09.2009 · Cited by 364
The signing brain: the neurobiology of sign language
MacSweeney, M., Capek, C. M., Campbell, R., & Woll, B. (2008) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2008.07.010 · Cited by 331
Mental imagery of speech and movement implicates the dynamics of internal forward models
Tian, X. (2010) · Frontiers in Psychology · doi:10.3389/fpsyg.2010.00166 · Cited by 259
“…can be induced by visual presentation of a face and that following auditory stimuli can be affected (see Jaaskelainen, 2010 for recent discussion of AV integration models; Sams et al., 1991; van Wassenhove et al., 2005; Von Kriegstein and Giraud, 2006; Skipper et al., 2007; Poeppel et al., 2008).”
Influential
Metadata
- Authors
- Skipper, J. I., van Wassenhove, V., Nusbaum, H. C., & Small, S. L.
- Venue
- Cerebral Cortex
- Publisher
- Oxford University Press (OUP)
- Published
- 2007-01-11
- Volume
- 17
- Issue
- 10
- Pages
- 2387-2399
- ISSN
- 1047-3211 (ISSN-L)
- 1460-2199
- OpenAlex
- W2102133930
- Cited by
- 369
- References
- 84
- FWCI
- 32.95
- Type
- journal article
- Language
- en
- Open access
- bronze
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.