Implicit multisensory associations influence voice recognition
Open accessAbstract
Natural objects provide partially redundant information to the brain through different sensory modalities. For example, voices and faces both give information about the speech content, age, and gender of a person. Thanks to this redundancy, multimodal recognition is fast, robust, and automatic. In unimodal perception, however, only part of the information about an object is available. Here, we addressed whether, even under conditions of unimodal sensory input, crossmodal neural circuits that have been shaped by previous associative learning become activated and underpin a performance benefit. We measured brain activity with functional magnetic resonance imaging before, while, and after participants learned to associate either sensory redundant stimuli, i.e. voices and faces, or arbitrary multimodal combinations, i.e. voices and written names, ring tones, and cell phones or brand names of these cell phones. After learning, participants were better at recognizing unimodal auditory voices that had been paired with faces than those paired with written names, and association of voices with faces resulted in an increased functional coupling between voice and face areas. No such effects were observed for ring tones that had been paired with cell phones or names. These findings demonstrate that brief exposure to ecologically valid and sensory redundant stimulus pairs, such as voices and faces, induces specific multisensory associations. Consistent with predictive coding theories, associative representations become thereafter available for unimodal perception and facilitate object recognition. These data suggest that for natural objects effective predictive signals can be generated across sensory systems and proceed by optimization of functional connectivity between specialized cortical sensory modules.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Network coordination facilitating perceptual processing, including speech comprehension, has been detected as elevated FC.
“Network coordination facilitating perceptual processing, including speech comprehension, has been detected as elevated FC (17,20,43–47).”
From Elevated functional connectivity along a corticostriatal loop and the mechanism of auditory/verbal hallucinations in patients with schizophrenia (2011) · cites this paper as [44]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Statistical parametric maps in functional imaging: A general linear approach
Friston, K. J., Holmes, A. P., Worsley, K. J., Poline, J. ‐., Frith, C. D., & Frackowiak, R. S. J. (1994) · Human Brain Mapping · doi:10.1002/hbm.460020402 · Cited by 9,688 · see citing works
The fusiform face area: a module in human extrastriate cortex specialized for face perception
Kanwisher, N., McDermott, J., & Chun, M. M. (1997) · The Journal of Neuroscience · doi:10.1523/jneurosci.17-11-04302.1997 · Cited by 7,738 · see citing works
Rao, R. P. N., & Ballard, D. H. (1999) · Nature Neuroscience · doi:10.1038/4580 · Cited by 5,483 · cited 1× · see citing works
“Models based on predictive coding assume that the brain captures the regularities of the natural world through statistical assessment of the properties of the environment [7,10,48].”
BackgroundHumans integrate visual and haptic information in a statistically optimal fashion
Ernst, M. O., & Banks, M. S. (2002) · Nature · doi:10.1038/415429a · Cited by 4,844 · see citing works
A theory of cortical responses.
Friston, K. (2005) · Philosophical Transactions of the Royal Society B: Biological Sciences · doi:10.1098/rstb.2005.1622 · Cited by 4,578 · see citing works
Understanding face recognition
Bruce, V., & Young, A. (1986) · British Journal of Psychology · doi:10.1111/j.2044-8295.1986.tb02199.x · Cited by 3,900 · see citing works
An internal model for sensorimotor integration
Wolpert, D. M., Ghahramani, Z., & Jordan, M. I. (1995) · Science · doi:10.1126/science.7569931 · Cited by 3,553 · see citing works
Psychophysiological and Modulatory Interactions in Neuroimaging
Friston, K. J., Buechel, C., Fink, G. R., Morris, J., Rolls, E., & Dolan, R. J. (1997) · NeuroImage · doi:10.1006/nimg.1997.0291 · Cited by 3,097 · see citing works
Mesulam, M. (1998) · Brain · doi:10.1093/brain/121.6.1013 · Cited by 3,072 · see citing works
Cohen, L., Dehaene, S., Naccache, L., Lehéricy, S., Dehaene-Lambertz, G., Hénaff, M.-A., & Michel, F. (2000) · Brain · doi:10.1093/brain/123.2.291 · Cited by 1,954 · see citing works
Voice-selective areas in human auditory cortex
Belin, P., Zatorre, R. J., Lafaille, P., Ahad, P. A., & Pike, G. B. (2000) · Nature · doi:10.1038/35002078 · Cited by 1,789 · see citing works
Is neocortex essentially multisensory?
GHAZANFAR, A., & SCHROEDER, C. (2006) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2006.04.008 · Cited by 1,463 · see citing works
Perception as Bayesian Inference
(1996) · doi:10.1017/cbo9780511984037 · Cited by 1,425 · see citing works
Expertise for cars and birds recruits brain areas involved in face recognition
Gauthier, I., Skudlarski, P., Gore, J. C., & Anderson, A. W. (2000) · Nature Neuroscience · doi:10.1038/72140 · Cited by 1,418 · cited 2× · see citing works
“Voicename association, however, balances voice-face learning, because matching names with voices is about as frequent in everyday life as matching faces with voices, and it additionally controls for familiarization with voices throughout the experiment and potential expertise effects [26,27].”
BackgroundGiard, M. H., & Peronnet, F. (1999) · Journal of Cognitive Neuroscience · doi:10.1162/089892999563544 · Cited by 1,072 · see citing works
Activation of Auditory Cortex During Silent Lipreading
Calvert, G. A., Bullmore, E. T., Brammer, M. J., Campbell, R., Williams, S., McGuire, P. K., Woodruff, P. W. R., Iversen, S. D., & David, A. S. (1997) · Science · doi:10.1126/science.276.5312.593 · Cited by 1,025 · see citing works
Anatomical evidence of multimodal integration in primate striate cortex
Falchier, A., Clavagnier, S., Barone, P., & Kennedy, H. (2002) · The Journal of Neuroscience · doi:10.1523/jneurosci.22-13-05749.2002 · Cited by 892 · cited 1× · see citing works
“Early crossmodal effects are also backed-up by anatomical studies that used retrograde labelling of monkey visual cortex to show the presence of projections arising from auditory cortex in several cortical layers [41–44].”
BackgroundVision as Bayesian inference: analysis by synthesis?
Yuille, A., & Kersten, D. (2006) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2006.05.002 · Cited by 849 · see citing works
Visual speech speeds up the neural processing of auditory speech
van Wassenhove, V., Grant, K. W., & Poeppel, D. (2005) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.0408949102 · Cited by 843 · see citing works
Temporal window of integration in auditory-visual speech perception
van Wassenhove, V., Grant, K. W., & Poeppel, D. (2006) · Neuropsychologia · doi:10.1016/j.neuropsychologia.2006.01.001 · Cited by 622 · see citing works
Understanding face recognition with an interactive activation model
Burton, A. M., Bruce, V., & Johnston, R. (1990) · British Journal of Psychology · doi:10.1111/j.2044-8295.1990.tb02367.x · Cited by 609 · see citing works
Probabilistic models of cognition: Conceptual foundations
Chater, N., Tenenbaum, J. B., & Yuille, A. (2006) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2006.05.007 · Cited by 546 · see citing works
Multisensory Integration of Dynamic Faces and Voices in Rhesus Monkey Auditory Cortex
Ghazanfar, A. A., Maier, J. X., Hoffman, K. L., & Logothetis, N. K. (2005) · Journal of Neuroscience · doi:10.1523/jneurosci.0799-05.2005 · Cited by 531 · see citing works
Neural Systems Underlying the Recognition of Familiar and Newly Learned Faces
Leveroni, C. L., Seidenberg, M., Mayer, A. R., Mead, L. A., Binder, J. R., & Rao, S. M. (2000) · Journal of Neuroscience · doi:10.1523/jneurosci.20-02-00878.2000 · Cited by 494 · see citing works
Where Bottom-up Meets Top-down: Neuronal Interactions during Perception and Imagery
Mechelli, A. (2004) · Cerebral Cortex · doi:10.1093/cercor/bhh087 · Cited by 480 · see citing works
The neural systems sustaining face and proper-name processing
Tempini, M. L. G. (1998) · Brain · doi:10.1093/brain/121.11.2103 · Cited by 465 · see citing works
Somatosensory Input to Auditory Association Cortex in the Macaque Monkey
Schroeder, C. E., Lindsley, R. W., Specht, C. M., Marcovici, A., Smiley, J. F., & Javitt, D. C. (2001) · Journal of Neurophysiology · doi:10.1152/jn.2001.85.3.1322 · Cited by 420 · see citing works
Quantitative association of vocal-tract and facial behavior
Yehia, H. C., Rubin, P. E., & Vatikiotis‐Bateson, E. (1998) · Speech Communication · doi:10.1016/s0167-6393(98)00048-x · Cited by 420 · see citing works
Knowledge of famous faces and names in semantic dementia.
Snowden, J., Thompson, J. C., & Neary, D. (2004) · Brain : a journal of neurology · doi:10.1093/brain/awh099 · Cited by 372 · see citing works
Cappe, C., & Barone, P. (2005) · European Journal of Neuroscience · doi:10.1111/j.1460-9568.2005.04462.x · Cited by 353 · see citing works
Reading Speech from Still and Moving Faces: The Neural Substrates of Visible Speech
Calvert, G. A., & Campbell, R. (2003) · Journal of Cognitive Neuroscience · doi:10.1162/089892903321107828 · Cited by 332 · see citing works
Visual Imagery of Famous Faces: Effects of Memory and Attention Revealed by fMRI
Ishai, A. (2002) · NeuroImage · doi:10.1006/nimg.2002.1330 · Cited by 331 · see citing works
Beyond faces and modularity: the power of an expertise framework
Bukach, C. M., Gauthier, L., & Tarr, M. J. (2006) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2006.02.004 · Cited by 330 · see citing works
Auditory Cortical Neurons Respond to Somatosensory Stimulation
Fu, K.-M. G., Johnston, T. A., Shah, A. S., Arnold, L. L., Smiley, J. F., Hackett, T. A., Garraghty, P. E., & Schroeder, C. E. (2003) · Journal of Neuroscience · doi:10.1523/jneurosci.23-20-07510.2003 · Cited by 317 · see citing works
Interaction of Face and Voice Areas during Speaker Recognition
von Kriegstein, K., Kleinschmidt, A. K., Sterzer, P., & Giraud, A. (2005) · Journal of Cognitive Neuroscience · doi:10.1162/0898929053279577 · Cited by 315 · see citing works
Self-citationThe processing and perception of size information in speech sounds
Smith, D. R. R., Patterson, R., Turner, R. E., Kawahara, H., & Irino, T. (2005) · Journal of the Acoustical Society of America · doi:10.1121/1.1828637 · Cited by 246 · see citing works
Principled and statistical connections in common sense conception
Prasada, S., & Dillingham, E. M. (2006) · Cognition · doi:10.1016/j.cognition.2005.01.003 · Cited by 230 · see citing works
Neural substrates for recognition of familiar voices: a PET study
Nakamura, K., Kawashima, R., Sugiura, M., Kato, T., Nakamura, A., Hatano, K., Nagumo, S., Kubota, K., Fukuda, H., Ito, K., et al. (2001) · Neuropsychologia · doi:10.1016/s0028-3932(01)00037-9 · Cited by 219 · see citing works
Gainotti, G., Barbier, A., & Marra, C. (2003) · Brain : a journal of neurology · doi:10.1093/brain/awg092 · Cited by 204 · see citing works
The role of multisensory memories in unisensory object discrimination
Lehmann, S., & Murray, M. M. (2005) · Cognitive Brain Research · doi:10.1016/j.cogbrainres.2005.02.005 · Cited by 194 · see citing works
`Putting the Face to the Voice'
Kamachi, M., Hill, H. C. H., Lander, K., & Vatikiotis‐Bateson, E. (2003) · Current Biology · doi:10.1016/j.cub.2003.09.005 · Cited by 187 · see citing works
Perceptual grouping and the interactions between visual cortical areas
Murray, S. O., Schrater, P., & Kersten, D. (2004) · Neural Networks · doi:10.1016/j.neunet.2004.03.010 · Cited by 149 · see citing works
Intra‐ and inter‐modal repetition priming of familiar faces and voices
Ellis, H. D., Jones, D. M., & Mosdell, N. (1997) · British Journal of Psychology · doi:10.1111/j.2044-8295.1997.tb02625.x · Cited by 145 · see citing works
Neuropsychological Impairments in the Recognition of Faces, Voices, and Personal Names
Neuner, F., & Schweinberger, S. R. (2000) · Brain and Cognition · doi:10.1006/brcg.1999.1196 · Cited by 117 · see citing works
Analysis of Neural Interactions Explains the Activation of Occipital Cortex by an Auditory Stimulus
McIntosh, A. R., Cabeza, R. E. L., & Lobaugh, N. J. (1998) · Journal of Neurophysiology · doi:10.1152/jn.1998.80.5.2790 · Cited by 105 · see citing works
Hanley, J. R., Smith, S., & Hadfield, J. (1998) · The Quarterly Journal of Experimental Psychology Section A · doi:10.1080/713755751 · Cited by 102 · see citing works
SIMULATING FACE RECOGNITION: IMPLICATIONS FOR MODELLING COGNITION
Young, A. W. (1999) · Cognitive Neuropsychology · doi:10.1080/026432999380960 · Cited by 90 · see citing works
Joassin, F., Maurage, P., Bruyer, R., Crommelinck, M., & Campanella, S. (2004) · Neuroscience Letters · doi:10.1016/j.neulet.2004.07.067 · Cited by 89 · see citing works
The brain uses single-trial multisensory memories to discriminate without awareness
Murray, M. M., Foxe, J. J., & Wylie, G. R. (2005) · NeuroImage · doi:10.1016/j.neuroimage.2005.04.016 · Cited by 88 · see citing works
Audiovisual speech facilitates voice learning
Sheffert, S. M., & Olson, E. (2004) · Perception & Psychophysics · doi:10.3758/bf03194884 · Cited by 84 · see citing works
Voice Recognition and Cross-Modal Responses to Familiar Speakers' Voices in Prosopagnosia
von Kriegstein, K., Kleinschmidt, A. K., & Giraud, A. (2005) · Cerebral Cortex · doi:10.1093/cercor/bhj073 · Cited by 83 · see citing works
Self-citationTsukiura, T., Mochizuki‐Kawai, H., & Fujii, T. (2005) · NeuroImage · doi:10.1016/j.neuroimage.2005.09.043 · Cited by 73 · see citing works
Introduction: a Bayesian formulation of visual perception
Knill, D. C., Kersten, D., & Yuille, A. (1996) · Cambridge University Press eBooks · Cited by 61 · see citing works
Why are Familiar-Only Experiences More Frequent for Voices than for Faces?
Hanley, J. R., & Turner, J. M. (2000) · The Quarterly Journal of Experimental Psychology Section A · doi:10.1080/713755942 · Cited by 53 · see citing works
Specification of cross-modal source information in isolated kinematic displays of speech
Lachs, L., & Pisoni, D. B. (2004) · The Journal of the Acoustical Society of America · doi:10.1121/1.1757454 · Cited by 47 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Whatever next? Predictive brains, situated agents, and the future of cognitive science
Clark, A. (2013) · Behavioral and Brain Sciences · doi:10.1017/s0140525x12000477 · Cited by 5,924 · see citing works
Crossmodal correspondences: A tutorial review
Spence, C. (2011) · Attention, Perception, & Psychophysics · doi:10.3758/s13414-010-0073-7 · Cited by 1,618 · see citing works
Benefits of multisensory learning
Shams, L., & Seitz, A. R. (2008) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2008.07.006 · Cited by 1,178 · see citing works
A Dual Role for Prediction Error in Associative Learning
den Ouden, H. E. M., Friston, K. J., Daw, N. D., McIntosh, A. R., & Stephan, K. E. (2009) · Cerebral Cortex · doi:10.1093/cercor/bhn161 · Cited by 306 · see citing works
Mental imagery of speech and movement implicates the dynamics of internal forward models
Tian, X. (2010) · Frontiers in Psychology · doi:10.3389/fpsyg.2010.00166 · Cited by 259 · see citing works
“Cumulatively, the results converge with audio-visual multisensory integration theories positing that an abstract auditory representation (auditory efference copy) can be induced by visual presentation of a face and that following auditory stimuli can be affected (see Jaaskelainen, 2010 for recent discussion of AV inte…”
Temporal predictive codes for spoken words in auditory cortex
Gagnepain, P., Henson, R. N., & Davis, M. H. (2012) · Current Biology · doi:10.1016/j.cub.2012.02.015 · Cited by 235 · see citing works
Neural mechanisms for voice recognition
Andics, A., McQueen, J. M., Petersson, K. M., Gál, V., Rudas, G., & Vidnyánszky, Z. (2010) · NeuroImage · doi:10.1016/j.neuroimage.2010.05.048 · Cited by 222 · see citing works
Tian, X., & Poeppel, D. (2012) · Frontiers in Human Neuroscience · doi:10.3389/fnhum.2012.00314 · Cited by 161 · see citing works
Hoffman, R. E., Fernandez, T., Pittman, B., & Hampson, M. (2011) · Biological Psychiatry · doi:10.1016/j.biopsych.2010.09.050 · Cited by 156 · see citing works
Memorable Audiovisual Narratives Synchronize Sensory and Supramodal Neural Responses
Cohen, S. S., & Parra, L. C. (2016) · eneuro · doi:10.1523/eneuro.0203-16.2016 · Cited by 138 · see citing works
Schelinski, S., Riedel, P., & von Kriegstein, K. (2014) · Neuropsychologia · doi:10.1016/j.neuropsychologia.2014.09.031 · Cited by 37 · see citing works
Self-citationWildgruber, D., Szameitat, D. P., Ethofer, T., Brück, C., Alter, K., Grodd, W., & Kreifelts, B. (2013) · PLoS ONE · doi:10.1371/journal.pone.0063441 · Cited by 33 · see citing works
Cross-modal prediction changes the timing of conscious access during the motion-induced blindness
Chang, A. Y.-C., Kanai, R., & Seth, A. K. (2015) · Consciousness and Cognition · doi:10.1016/j.concog.2014.11.005 · Cited by 25 · see citing works
Multisensory Facilitation of Working Memory Training
Pahor, A., Collins, C., Smith-Peirce, R. N., Moon, A., Stavropoulos, T., Silva, I., Peng, E., Jaeggi, S. M., & Seitz, A. R. (2021) · Journal of Cognitive Enhancement · doi:10.1007/s41465-020-00196-y · Cited by 17 · see citing works
- Authors
- von Kriegstein, K., & Giraud, A.-L.
- Affiliations
- Goethe University Frankfurt (DE)
- ORCID
- Giraud, Anne-Lise
- Venue
- PLoS Biology
- Publisher
- Public Library of Science (PLoS)
- Published
- 2006-09-26
- Volume
- 4
- Issue
- 10
- Pages
- e326
- ISSN
- 1544-9173 (ISSN-L)
- 1545-7885
- PMC
- PMC1570760
- OpenAlex
- W2142393318
- Cited by
- 309
- References
- 55
- Citations by year
- 2026: 6 · 2025: 4 · 2024: 8 · 2023: 14 · 2022: 22 · 2021: 13 · 2020: 16 · 2019: 14 · 2018: 24 · 2017: 19 · 2016: 10 · 2015: 12 · 2014: 27 · 2013: 17 · 2012: 21
- FWCI
- 13.82
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Locations
- PLoS Biology (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, cc-by)
- PubMed (publishedVersion)
- HAL (submittedVersion, other-oa)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Max Planck Digital Library (submittedVersion, other-oa)
- Figshare (submittedVersion, cc-by)
- Publication Server of Goethe University Frankfurt am Main (Goethe University Frankfurt) (submittedVersion, cc-by)
- MPG.PuRe (Max Planck Society) (submittedVersion, other-oa)
- Funders
- Bundesministerium für Bildung und Forschung
- Topics
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Multisensory perception and integration
- Life Sciences › Neuroscience › Cognitive Neuroscience › Tactile and Sensory Interactions
- Life Sciences › Neuroscience › Cognitive Neuroscience › Face Recognition and Perception
- Keywords
- voice recognition, face recognition (cognition), sensory redundancy, functional connectivity, predictive coding, audiovisual integration, object recognition, brand names
- MeSH terms
- Acoustic Stimulation, Association, Behavior, Brain, Female, Humans, Learning, Magnetic Resonance Imaging, Male, Neurons, Pattern Recognition, Visual, Perception, Surveys and Questionnaires, Voice, Recognition, Psychology
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
