The DOI matches its Crossref record; 89 of the 89 references it cites were found in a registry or an index.
Perceptual Fusion and Stimulus Coincidence in the Cross-Modal Integration of Speech
Open accessAbstract
Human speech perception is profoundly influenced by vision. Watching a speaker's mouth movements significantly improves comprehension, both for normal listeners in noisy environments and especially for the hearing impaired. A number of brain regions have been implicated in audiovisual speech tasks, but little evidence distinguishes them functionally. In an event-related functional magnetic resonance imaging study, we differentiate neural systems that evaluate cross-modal coincidence of the physical stimuli from those that mediate perceptual binding. Regions consistently involved in perceptual fusion per se included Heschl's gyrus, superior temporal sulcus, middle intraparietal sulcus, and inferior frontal gyrus. Successful fusion elicited activity biased toward the left hemisphere, although failed cross-modal binding recruited regions in both hemispheres. A broad network of other areas, including the superior colliculus, anterior insula, and anterior intraparietal sulcus, were more involved with evaluating the spatiotemporal correspondence of speech stimuli, regardless of a subject's perception. All of these showed greater activity to temporally offset stimuli than to audiovisually synchronous stimuli. Our results demonstrate how elements of the cross-modal speech integration network differ in their sensitivity to physical reality versus perceptual experience.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Separate visual pathways for perception and action
Goodale, M. A., & Milner, A. D. (1992) · Trends in Neurosciences · doi:10.1016/0166-2236(92)90344-8 · Cited by 6,519
Hearing lips and seeing voices
MCGURK, H., & MACDONALD, J. (1976) · Nature · doi:10.1038/264746a0 · Cited by 6,183
Premotor cortex and the recognition of motor actions
Rizzolatti, G., Fadiga, L., Gallese, V., & Fogassi, L. (1996) · Cognitive Brain Research · doi:10.1016/0926-6410(95)00038-0 · Cited by 4,870
Anterior Cingulate Conflict Monitoring and Adjustments in Control
Kerns, J. G., Cohen, J. D., MacDonald, A. W., Cho, R. Y., Stenger, V. A., & Carter, C. S. (2004) · Science · doi:10.1126/science.1089910 · Cited by 2,891
The motor theory of speech perception revised
Liberman, A. M., & Mattingly, I. G. (1985) · Cognition · doi:10.1016/0010-0277(85)90021-6 · Cited by 2,861
Rizzolatti, G., & Arbib, M. A. (1998) · Trends in Neurosciences · doi:10.1016/s0166-2236(98)01260-0 · Cited by 2,858
Visual Contribution to Speech Intelligibility in Noise
Sumby, W. H., & Pollack, I. (1954) · The Journal of the Acoustical Society of America · doi:10.1121/1.1907309 · Cited by 2,583
Hickok, G., & Poeppel, D. (2004) · Cognition · doi:10.1016/j.cognition.2003.10.011 · Cited by 2,212
Neural Basis of a Perceptual Decision in the Parietal Cortex (Area LIP) of the Rhesus Monkey
Shadlen, M. N., & Newsome, W. T. (2001) · Journal of Neurophysiology · doi:10.1152/jn.2001.86.4.1916 · Cited by 1,755
Poldrack, R. A., Wagner, A. D., Prull, M. W., Desmond, J. E., Glover, G. H., & Gabrieli, J. D. E. (1999) · NeuroImage · doi:10.1006/nimg.1999.0441 · Cited by 1,448
Human temporal lobe activation by speech and nonspeech sounds
Binder, J. R. (2000) · Cerebral Cortex · doi:10.1093/cercor/10.5.512 · Cited by 1,345
Intentional Maps in Posterior Parietal Cortex
Andersen, R. A., & Buneo, C. A. (2002) · Annual Review of Neuroscience · doi:10.1146/annurev.neuro.25.112701.142922 · Cited by 1,294
A new brain region for coordinating speech articulation
Dronkers, N. F. (1996) · Nature · doi:10.1038/384159a0 · Cited by 1,192
Picard, N., & Strick, P. L. (2001) · Current Opinion in Neurobiology · doi:10.1016/s0959-4388(01)00266-5 · Cited by 1,191
Identification of a pathway for intelligible speech in the left temporal lobe
Scott, S. K. (2000) · Brain · doi:10.1093/brain/123.12.2400 · Cited by 1,160
THE ANATOMY OF PHONOLOGICAL AND SEMANTIC PROCESSING IN NORMAL SUBJECTS
DÉMONET, J.-F., CHOLLET, F., RAMSAY, S., CARDEBAT, D., NESPOULOUS, J.-L., WISE, R., RASCOL, A., & FRACKOWIAK, R. (1992) · Brain · doi:10.1093/brain/115.6.1753 · Cited by 1,154
Geschwind, N. (1970) · Science · doi:10.1126/science.170.3961.940 · Cited by 1,132
Giard, M. H., & Peronnet, F. (1999) · Journal of Cognitive Neuroscience · doi:10.1162/089892999563544 · Cited by 1,072
Crossmodal Processing in the Human Brain: Insights from Functional Neuroimaging Studies
Calvert, G. A. (2001) · Cerebral Cortex · doi:10.1093/cercor/11.12.1110 · Cited by 1,012
Subdivisions of auditory cortex and processing streams in primates
Kaas, J. H., & Hackett, T. A. (2000) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.97.22.11793 · Cited by 996
Calvert, G. A., Campbell, R., & Brammer, M. J. (2000) · Current Biology · doi:10.1016/s0960-9822(00)00513-3 · Cited by 954
Mental Rotation of the Neuronal Population Vector
Georgopoulos, A. P., Lurito, J. T., Petrides, M., Schwartz, A. B., & Massey, J. T. (1989) · Science · doi:10.1126/science.2911737 · Cited by 907
Polymodal Motion Processing in Posterior Parietal and Premotor Cortex
Bremmer, F., Schlack, A., Shah, N. J., Zafiris, O., Kubischik, M., Hoffmann, K.-P., Zilles, K., & Fink, G. R. (2001) · Neuron · doi:10.1016/s0896-6273(01)00198-2 · Cited by 803
Hierarchical Processing in Spoken Language Comprehension
Davis, M. H., & Johnsrude, I. S. (2003) · The Journal of Neuroscience · doi:10.1523/jneurosci.23-08-03423.2003 · Cited by 772
Ferrari, P. F., Gallese, V., Rizzolatti, G., & Fogassi, L. (2003) · European Journal of Neuroscience · doi:10.1046/j.1460-9568.2003.02601.x · Cited by 728
Motion perception: seeing and deciding.
Shadlen, M. N., & Newsome, W. T. (1996) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.93.2.628 · Cited by 723
Auditory-motor interaction revealed by fMRI: speech, music, and working memory in area Spt
Hickok, G., Buchsbaum, B., Humphries, C., & Muftuler, T. (2003) · Journal of Cognitive Neuroscience · doi:10.1162/089892903322307393 · Cited by 700
Electrophysiology and brain imaging of biological motion
Puce, A., & Perrett, D. (2003) · Philosophical Transactions of the Royal Society of London. Series B: Biological Sciences · doi:10.1098/rstb.2002.1221 · Cited by 693
Visual illusion induced by sound
Shams, L., Kamitani, Y., & Shimojo, S. (2002) · Cognitive Brain Research · doi:10.1016/s0926-6410(02)00069-1 · Cited by 657
Shmuel, A., Yacoub, E., Pfeuffer, J., Van de Moortele, P.-F., Adriany, G., Hu, X., & Ugurbil, K. (2002) · Neuron · doi:10.1016/s0896-6273(02)01061-9 · Cited by 631
Modulation of Human Visual Cortex by Crossmodal Spatial Attention
Macaluso, E., Frith, C. D., & Driver, J. (2000) · Science · doi:10.1126/science.289.5482.1206 · Cited by 613
Architecture and Connections of Cortical Association Areas
Pandya, D. N., & Yeterian, E. H. (1985) · Cerebral Cortex · doi:10.1007/978-1-4757-9619-3_1 · Cited by 584
Hierarchical and asymmetric temporal sensitivity in human auditory cortices
Boemio, A., Fromm, S., Braun, A., & Poeppel, D. (2005) · Nature Neuroscience · doi:10.1038/nn1409 · Cited by 582
Separate neural subsystems within ‘Wernicke's area’
Wise, R. J. S., Scott, S. K., Blank, S. C., Mummery, C. J., Murphy, K., & Warburton, E. A. (2001) · Brain · doi:10.1093/brain/124.1.83 · Cited by 572
The use of visible speech cues for improving auditory detection of spoken sentences
Grant, K. W., & Seitz, P.-F. (2000) · The Journal of the Acoustical Society of America · doi:10.1121/1.1288668 · Cited by 521
Burock, M. A., Buckner, R. L., Woldorff, M. G., Rosen, B. R., & Dale, A. M. (1998) · NeuroReport · doi:10.1097/00001756-199811160-00030 · Cited by 517
Multimodal integration for the representation of space in the posterior parietal cortex
Andersen, R. A. (1997) · Philosophical Transactions of the Royal Society of London. Series B: Biological Sciences · doi:10.1098/rstb.1997.0128 · Cited by 515
Unraveling multisensory integration: patchy organization within human STS multisensory cortex
Beauchamp, M. S., Argall, B. D., Bodurka, J., Duyn, J. H., & Martin, A. (2004) · Nature Neuroscience · doi:10.1038/nn1333 · Cited by 513
Neural correlates of sensory and decision processes in auditory object identification
Binder, J. R., Liebenthal, E., Possing, E. T., Medler, D. A., & Ward, B. D. (2004) · Nature Neuroscience · doi:10.1038/nn1198 · Cited by 482
Calvert, G. A., Hansen, P. C., Iversen, S. D., & Brammer, M. J. (2001) · NeuroImage · doi:10.1006/nimg.2001.0812 · Cited by 464
Auditory belt and parabelt projections to the prefrontal cortex in the Rhesus monkey
Romanski, L. M., Bates, J. F., & Goldman-Rakic, P. S. (1999) · The Journal of Comparative Neurology · doi:10.1002/(sici)1096-9861(19990111)403:2<141::aid-cne1>3.0.co;2-v · Cited by 462
Human temporal-lobe response to vocal sounds
Belin, P., Zatorre, R. J., & Ahad, P. (2002) · Cognitive Brain Research · doi:10.1016/s0926-6410(01)00084-2 · Cited by 452
Separating Processes within a Trial in Event-Related Functional MRI
Ollinger, J. M., Corbetta, M., & Shulman, G. L. (2001) · NeuroImage · doi:10.1006/nimg.2000.0711 · Cited by 445
Quantifying the contribution of vision to speech perception in noise
Macleod, A., & Summerfield, Q. (1987) · British Journal of Audiology · doi:10.3109/03005368709077786 · Cited by 426
PET Studies of Phonetic Processing of Speech: Review, Replication, and Reanalysis
Zatorre, R. J., Meyer, E., Gjedde, A., & Evans, A. C. (1996) · Cerebral Cortex · doi:10.1093/cercor/6.1.21 · Cited by 422
Reaction time as a measure of intersensory facilitation.
Hershenson, M. (1962) · Journal of Experimental Psychology · doi:10.1037/h0039516 · Cited by 418
Motor Area Activity During Mental Rotation Studied by Time-Resolved Single-Trial fMRI
Richter, W., Somorjai, R., Summers, R., Jarmasz, M., Menon, R. S., Gati, J. S., Georgopoulos, A. P., Tegeler, C., Ugurbil, K., & Kim, S.-G. (2000) · Journal of Cognitive Neuroscience · doi:10.1162/089892900562129 · Cited by 416
The insula (Island of Reil) and its role in auditory processing
Bamiou, D.-E., Musiek, F. E., & Luxon, L. M. (2003) · Brain Research Reviews · doi:10.1016/s0165-0173(03)00172-3 · Cited by 397
Stein, B. E., Meredith, M. A., Huneycutt, W. S., & McDade, L. (1989) · Journal of Cognitive Neuroscience · doi:10.1162/jocn.1989.1.1.12 · Cited by 394
The Detection of Auditory Visual Desynchrony
Dixon, N. F., & Spitz, L. (1980) · Perception · doi:10.1068/p090719 · Cited by 390
Separating Processes within a Trial in Event-Related Functional MRI
Ollinger, J. M., Shulman, G. L., & Corbetta, M. (2001) · NeuroImage · doi:10.1006/nimg.2000.0710 · Cited by 385
Neural Correlates of Auditory–Visual Stimulus Onset Asynchrony Detection
Bushara, K. O., Grafman, J., & Hallett, M. (2001) · The Journal of Neuroscience · doi:10.1523/jneurosci.21-01-00300.2001 · Cited by 361
Temporal constraints on the McGurk effect
Munhall, K. G., Gribble, P., Sacco, L., & Ward, M. (1996) · Perception & Psychophysics · doi:10.3758/bf03206811 · Cited by 358
Nishitani, N., & Hari, R. (2002) · Neuron · doi:10.1016/s0896-6273(02)01089-9 · Cited by 353
Grant, K. W., Walden, B. E., & Seitz, P. F. (1998) · The Journal of the Acoustical Society of America · doi:10.1121/1.422788 · Cited by 342
Response amplification in sensory-specific cortices during crossmodal binding
Calvert, G. A., Brammer, M. J., Bullmore, E. T., Campbell, R., Iversen, S. D., & David, A. S. (1999) · NeuroReport · doi:10.1097/00001756-199908200-00033 · Cited by 331
D’Esposito, M., Postle, B. R., Jonides, J., & Smith, E. E. (1999) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.96.13.7514 · Cited by 329
Self-citationSpatial and temporal factors determine auditory-visual interactions in human saccadic eye movements
Frens, M. A., Van Opstal, A. J., & Van Der Willigen, R. F. (1995) · Perception & Psychophysics · doi:10.3758/bf03206796 · Cited by 326
Lateralization of speech and auditory temporal processing
Belin, P., Zilbovicius, M., Crozier, S., Thivard, L., Fontaine, A. A., Masure, M.-C., & Samson, Y. (1998) · Journal of Cognitive Neuroscience · doi:10.1162/089892998562834 · Cited by 324
Primary auditory cortex activation by visual speech: an fMRI study at 3???T
Pekkola, J., Ojanen, V., Autti, T., J????skel??inen, I. P., M??tt??nen, R., Tarkiainen, A., & Sams, M. (2005) · NeuroReport · doi:10.1097/00001756-200502080-00010 · Cited by 306
Detection Power, Estimation Efficiency, and Predictability in Event-Related fMRI
Liu, T. T., Frank, L. R., Wong, E. C., & Buxton, R. B. (2001) · NeuroImage · doi:10.1006/nimg.2000.0728 · Cited by 261
Visual-auditory interactions in sensorimotor processing: Saccades versus manual responses.
Hughes, H. C., Reuter‐Lorenz, P. A., Nozawa, G., & Fendrich, R. (1994) · Journal of Experimental Psychology Human Perception & Performance · doi:10.1037//0096-1523.20.1.131 · Cited by 252
Auditory-visual speech perception examined by fMRI and PET
Sekiyama, K., Kanno, I., Miura, S., & Sugita, Y. (2003) · Neuroscience Research · doi:10.1016/s0168-0102(03)00214-1 · Cited by 249
Tracking the Mind's Image in the Brain I
Formisano, E., Linden, D. E. J., Di Salle, F., Trojano, L., Esposito, F., Sack, A. T., Grossi, D., Zanella, F. E., & Goebel, R. (2002) · Neuron · doi:10.1016/s0896-6273(02)00747-x · Cited by 240
Spatial and temporal factors during processing of audiovisual speech: a PET study
Macaluso, E., George, N., Dolan, R., Spence, C., & Driver, J. (2004) · NeuroImage · doi:10.1016/j.neuroimage.2003.09.049 · Cited by 237
Microstimulation of visual cortex affects the speed of perceptual decisions
Ditterich, J., Mazurek, M. E., & Shadlen, M. N. (2003) · Nature Neuroscience · doi:10.1038/nn1094 · Cited by 234
Neural correlates of cross-modal binding
Bushara, K. O., Hanakawa, T., Immisch, I., Toma, K., Kansaku, K., & Hallett, M. (2003) · Nature Neuroscience · doi:10.1038/nn993 · Cited by 226
Giraud, A. L. (2004) · Cerebral Cortex · doi:10.1093/cercor/bhg124 · Cited by 213
Intermodal timing relations and audio-visual speech recognition by normal-hearing adults
McGrath, M., & Summerfield, Q. (1985) · The Journal of the Acoustical Society of America · doi:10.1121/1.392336 · Cited by 201
Neural processes underlying perceptual enhancement by visual speech gestures
Callan, D. E., Jones, J. A., Munhall, K., Callan, A. M., Kroos, C., & Vatikiotis-Bateson, E. (2003) · NeuroReport · doi:10.1097/00001756-200312020-00016 · Cited by 190
Auditory psychomotor coordination and visual search performance
Perrott, D. R., Saberi, K., Brown, K., & Strybel, T. Z. (1990) · Perception & Psychophysics · doi:10.3758/bf03211521 · Cited by 178
Processing of audiovisual speech in Broca's area
Ojanen, V., Möttönen, R., Pekkola, J., Jääskeläinen, I. P., Joensuu, R., Autti, T. H., & Sams, M. E. (2005) · NeuroImage · doi:10.1016/j.neuroimage.2004.12.001 · Cited by 178
Klucharev, V., Möttönen, R., & Sams, M. (2003) · Cognitive Brain Research · doi:10.1016/j.cogbrainres.2003.09.004 · Cited by 174
A comparison of bound and unbound audio–visual information processing in the human cerebral cortex
Olson, I. R., Gatenby, J. C., & Gore, J. C. (2002) · Cognitive Brain Research · doi:10.1016/s0926-6410(02)00067-8 · Cited by 172
Parietal activity and the perceived direction of ambiguous apparent motion
Williams, Z. M., Elfar, J. C., Eskandar, E. N., Toth, L. J., & Assad, J. A. (2003) · Nature Neuroscience · doi:10.1038/nn1055 · Cited by 169
Effects of Phonetic Context on Audio-Visual Intelligibility of French
Benoı̂t, C., Mohamadi, T., & Kandel, S. (1994) · Journal of Speech Language and Hearing Research · doi:10.1044/jshr.3705.1195 · Cited by 163
Banati, R. B., Goerres, G. W., Tjoa, C., Aggleton, J. P., & Grasby, P. (2000) · Neuropsychologia · doi:10.1016/s0028-3932(99)00074-3 · Cited by 149
Two Corticotectal Areas Facilitate Multisensory Orientation Behavior
Jiang, W., Jiang, H., & Stein, B. E. (2002) · Journal of Cognitive Neuroscience · doi:10.1162/089892902760807230 · Cited by 142
Callan, D. E., Jones, J. A., Munhall, K. G., Kroos, C., Callan, A. M., & Vatikiotis‐Bateson, E. (2004) · Journal of Cognitive Neuroscience · doi:10.1162/089892904970771 · Cited by 133
Callan, D. E., Callan, A. M., Kroos, C., & Vatikiotis‐Bateson, E. (2001) · Cognitive Brain Research · doi:10.1016/s0926-6410(00)00054-9 · Cited by 107
Perrott, D. R., Sadralodabai, T., Saberi, K., & Strybel, T. Z. (1991) · Human Factors The Journal of the Human Factors and Ergonomics Society · doi:10.1177/001872089103300402 · Cited by 98
The parieto‐collicular pathway: anatomical location and contribution to saccade generation
Gaymard, B. M., Lynch, J. C., Ploner, C. J., Condy, C., & Rivaud-Péchoux, S. (2003) · European Journal of Neuroscience · doi:10.1046/j.1460-9568.2003.02570.x · Cited by 95
Möttönen, R., Schürmann, M., & Sams, M. E. (2004) · Neuroscience Letters · doi:10.1016/j.neulet.2004.03.076 · Cited by 91
SPEECH INTELLIGIBILITY DERIVED FROM ASYNCHRONOUS PROCESSING OF AUDITORY-VISUAL INFORMATION
Grant, K. W., & Greenberg, S. (2001) · AVSP · Cited by 90
Evidence for training-induced crossmodal reorganization of cortical functions in trumpet players
Schulz, M., Roß, B., & Pantev, C. (2003) · Neuroreport · doi:10.1097/00001756-200301200-00029 · Cited by 86
Book Review: Cortex Governs Multisensory Integration in the Midbrain
Stein, B. E., Wallace, M. W., Stanford, T. R., & Jiang, W. (2002) · The Neuroscientist · doi:10.1177/107385840200800406 · Cited by 74
Temporal Characteristics of Sensory Interaction in Choice Reaction Times.
Morrell, L. K. (1968) · Journal of Experimental Psychology · doi:10.1037/h0021289 · Cited by 59
Chapter 10 Nonvisual influences on visual-information processing in the superior colliculus
Stein, B. E., Jiang, W., Wallace, M. T., & Stanford, T. R. (2001) · Progress in brain research · doi:10.1016/s0079-6123(01)34011-6 · Cited by 35
Worsley (1996) · doi:10.1002/(sici)1097-0193(1996)4:1
Cited by
Driver, J., & Noesselt, T. (2008) · Neuron · doi:10.1016/j.neuron.2007.12.013 · Cited by 940
Integrating face and voice in person perception
Campanella, S., & Belin, P. (2007) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2007.10.001 · Cited by 527
Multisensory temporal integration in autism spectrum disorders
Stevenson, R. A., Siemann, J. K., Schneider, B. C., Eberly, H. E., Woynaroski, T. G., Camarata, S. M., & Wallace, M. T. (2014) · The Journal of Neuroscience · doi:10.1523/jneurosci.3615-13.2014 · Cited by 482
The Representation of Tool Use in Humans and Monkeys: Common and Uniquely Human Features
Peeters, R., Simone, L., Nelissen, K., Fabbri-Destro, M., Vanduffel, W., Rizzolatti, G., & Orban, G. A. (2009) · The Journal of Neuroscience · doi:10.1523/jneurosci.2040-09.2009 · Cited by 364
Multisensory Processing in Review: from Physiology to Behaviour
Alais, D., Newell, F., & Mamassian, P. (2010) · Seeing and Perceiving · doi:10.1163/187847510x488603 · Cited by 324
Metadata
- Authors
- Miller, L. M., & D'Esposito, M.
- Venue
- The Journal of Neuroscience
- Publisher
- Society for Neuroscience
- Published
- 2005-06-22
- Volume
- 25
- Issue
- 25
- Pages
- 5884-5893
- ISSN
- 0270-6474 (ISSN-L)
- 1529-2401
- OpenAlex
- W2144573398
- Cited by
- 322
- References
- 89
- FWCI
- 15.41
- Type
- journal article
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.