Predictive coding and multisensory integration: an attentional account of the multisensory mind
Open accessThe DOI matches its Crossref record; 29 of the 29 references it cites were found in a registry or an index.
Abstract
Multisensory integration involves a host of different cognitive processes, occurring at different stages of sensory processing. Here I argue that, despite recent insights suggesting that multisensory interactions can occur at very early latencies, the actual integration of individual sensory traces into an internally consistent mental representation is dependent on both top-down and bottom-up processes. Moreover, I argue that this integration is not limited to just sensory inputs, but that internal cognitive processes also shape the resulting mental representation. Studies showing that memory recall is affected by the initial multisensory context in which the stimuli were presented will be discussed, as well as several studies showing that mental imagery can affect multisensory illusions. This empirical evidence will be discussed from a predictive coding perspective, in which a central top-down attentional process is proposed to play a central role in coordinating the integration of all these inputs into a coherent mental representation.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Control of goal-directed and stimulus-driven attention in the brain
Corbetta, M., & Shulman, G. L. (2002) · Nature Reviews Neuroscience · doi:10.1038/nrn755 · Cited by 12,881
The free-energy principle: a unified brain theory?
Friston, K. (2010) · Nature Reviews Neuroscience · doi:10.1038/nrn2787 · Cited by 8,120 · cited 2×
“The predictive coding framework states that the brain produces a Bayesian estimate of the environment (Friston, 2010).”
BackgroundCognitive maps in rats and men
Tolman, E. C. (1948) · Psychological Review · doi:10.1037/h0061626 · Cited by 6,641
Place navigation impaired in rats with hippocampal lesions
Morris, R. G. M., Garrud, P., Rawlins, J. N. P., & O'Keefe, J. (1982) · Nature · doi:10.1038/297681a0 · Cited by 6,340 · cited 2×
“Moreover, spatial and temporal proximity (Lewald and Guski, 2003), or the relative intensity (known as the “law of inverse effectiveness”; see Holmes, 2009) of multisensory inputs may be critical factors in determining whether two inputs integrate.”
BackgroundHearing lips and seeing voices
MCGURK, H., & MACDONALD, J. (1976) · Nature · doi:10.1038/264746a0 · Cited by 6,183
Rao, R. P. N., & Ballard, D. H. (1999) · Nature Neuroscience · doi:10.1038/4580 · Cited by 5,483 · cited 1×
“For instance, Rao and Ballard (1999) presented a predictive coding model of receptive field effects in striate cortex and stated that, in their simulations, the “internal model is encoded in a distributed manner within the synapses of model neurons at each level.””
BackgroundHumans integrate visual and haptic information in a statistically optimal fashion
Ernst, M. O., & Banks, M. S. (2002) · Nature · doi:10.1038/415429a · Cited by 4,844
Perception of the speech code.
Liberman, A. M., Cooper, F. S., Shankweiler, D. P., & Studdert-Kennedy, M. (1967) · Psychological Review · doi:10.1037/h0020279 · Cited by 3,722 · cited 2×
“Speech signal segmentation is generally problematic (specifically under noisy conditions; see Ross et al., 2007; Foxe et al., 2015), because there is only a very loose connection between speech sounds and the underlying phoneme structure (Liberman et al., 1967).”
BackgroundThe Ventriloquist Effect Results from Near-Optimal Bimodal Integration
Alais, D., & Burr, D. (2004) · Current Biology · doi:10.1016/j.cub.2004.01.029 · Cited by 1,936 · cited 1×
“Additionally, it has been successfully applied to explain a number of basic multisensory processing phenomena (Ernst and Banks, 2002; Alais and Burr, 2004; Helbig and Ernst, 2007; Holmes, 2009).”
BackgroundIs neocortex essentially multisensory?
GHAZANFAR, A., & SCHROEDER, C. (2006) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2006.04.008 · Cited by 1,463
Immediate perceptual response to intersensory discrepancy.
Welch, R. B., & Warren, D. H. (1980) · Psychological Bulletin · doi:10.1037/0033-2909.88.3.638 · Cited by 1,176 · cited 2×
“Additionally, it has been shown that a stimulus presented in one modality can affect the processing of an accessory stimulus presented in another modality, either due to its task relevance (Busse et al., 2005; Donohue et al., 2014), or because of learned associations between the individual inputs…”
BackgroundGiard, M. H., & Peronnet, F. (1999) · Journal of Cognitive Neuroscience · doi:10.1162/089892999563544 · Cited by 1,072
Causal Inference in Multisensory Perception
Körding, K. P., Beierholm, U., Ma, W. J., Quartz, S., Tenenbaum, J. B., & Shams, L. (2007) · PLoS ONE · doi:10.1371/journal.pone.0000943 · Cited by 1,069 · cited 2×
“For instance Wallace et al. (2004) and Körding et al. (2007) have shown that even when low-level features of multisensory stimuli are perfectly matched, behavioral performance is impaired when these features are perceived to originate from two separate sources.”
BackgroundCognitive control, hierarchy, and the rostro–caudal organization of the frontal lobes
Badre, D. (2008) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2008.02.004 · Cited by 980
Calvert, G. A., Campbell, R., & Brammer, M. J. (2000) · Current Biology · doi:10.1016/s0960-9822(00)00513-3 · Cited by 954
Anatomical evidence of multimodal integration in primate striate cortex
Falchier, A., Clavagnier, S., Barone, P., & Kennedy, H. (2002) · The Journal of Neuroscience · doi:10.1523/jneurosci.22-13-05749.2002 · Cited by 892 · cited 2×
“These pathways are thought to connect the sensory cortices, either directly to each other (Falchier et al., 2002), or through cortico-thalamic-cortical pathways (Hackett et al., 2007; Lakatos et al., 2007; van den Brink et al., 2014), suggesting that information can be transferred relatively…”
BackgroundThe multifaceted interplay between attention and multisensory integration.
Talsma, D., Senkowski, D., Soto-Faraco, S., & Woldorff, M. G. (2010) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2010.06.008 · Cited by 828
Self-citationSpeech perception without traditional speech cues
Remez, R. E., Rubin, P. E., Pisoni, D. B., & Carrell, T. D. (1981) · Science · doi:10.1126/science.7233191 · Cited by 800
Less Is More: Expectation Sharpens Representations in the Primary Visual Cortex
Kok, P., Jehee, J. F. M., & de Lange, F. P. (2012) · Neuron · doi:10.1016/j.neuron.2012.04.034 · Cited by 798
Pip and pop: Nonspatial auditory signals improve spatial visual search.
Van der Burg, E., Olivers, C. N. L., Bronkhorst, A. W., & Theeuwes, J. (2008) · Journal of Experimental Psychology: Human Perception and Performance · doi:10.1037/0096-1523.34.5.1053 · Cited by 485
Talsma, D., Doty, T. J., & Woldorff, M. G. (2006) · Cerebral Cortex · doi:10.1093/cercor/bhk016 · Cited by 434 · cited 3×
“This point thus illustrates that prior experience and background knowledge may influence multisensory processing; a topic that will be discussed in more detail below.”
BackgroundSelf-citationAttention reverses the effect of prediction in silencing sensory signals
Kok, P., Rahnev, D., Jehee, J. F. M., Lau, H. C., & de Lange, F. P. (2012) · Cerebral Cortex · doi:10.1093/cercor/bhr310 · Cited by 430
Multisensory integration: methodological approaches and emerging principles in the human brain.
Calvert, G. A., & Thesen, T. (2004) · Journal of Physiology-Paris · doi:10.1016/j.jphysparis.2004.03.018 · Cited by 425 · cited 2×
“This crossfeeding may modify the original input signal and can therefore be described as a multisensory interaction, but not necessarily as multisensory integration.”
BackgroundLearning to integrate arbitrary signals from vision and touch
Ernst, M. O. (2007) · Journal of Vision · doi:10.1167/7.5.7 · Cited by 332
Decoding sound and imagery content in early visual cortex
Vetter, P., Smith, F. W., & Muckli, L. (2014) · Current Biology · doi:10.1016/j.cub.2014.04.020 · Cited by 323
Foxe, J. J., Molholm, S., Del Bene, V. A., Frey, H.-P., Russo, N. N., Blanco, D., Saint-Amour, D., & Ross, L. A. (2015) · Cerebral Cortex · doi:10.1093/cercor/bht213 · Cited by 263
Bayesian multisensory integration and cross-modal spatial links.
Deneve, S., & Pouget, A. (2004) · Journal of Physiology-Paris · doi:10.1016/j.jphysparis.2004.03.011 · Cited by 230
Looming Signals Reveal Synergistic Principles of Multisensory Integration
Cappe, C., Thelen, A., Romei, V., Thut, G., & Murray, M. M. (2012) · The Journal of Neuroscience · doi:10.1523/jneurosci.5517-11.2012 · Cited by 107
Auditory, tactile, and multisensory cues facilitate search for dynamic visual stimuli
Ngo, M. K., & Spence, C. (2010) · Attention, Perception, & Psychophysics · doi:10.3758/app.72.6.1654 · Cited by 94
Cited by
Neuroscience of virtual reality: from virtual exposure to embodied medicine
Riva, G., Wiederhold, B. K., & Mantovani, F. (2019) · Cyberpsychology, Behavior, and Social Networking · doi:10.1089/cyber.2017.29099.gri · Cited by 608
Senses of place: architectural design for the multisensory mind
Spence, C. (2020) · Cognitive Research: Principles and Implications · doi:10.1186/s41235-020-00243-4 · Cited by 380
Riva, G., & Dakanalis, A. (2018) · Frontiers in Human Neuroscience · doi:10.3389/fnhum.2018.00049 · Cited by 103
To integrate or not to integrate: Temporal dynamics of hierarchical Bayesian causal inference
Aller, M., & Noppeney, U. (2019) · PLOS Biology · doi:10.1371/journal.pbio.3000210 · Cited by 98
Oral referral: On the mislocalization of odours to the mouth
Spence, C. (2016) · Food Quality and Preference · doi:10.1016/j.foodqual.2016.02.006 · Cited by 80
Metadata
- Authors
- Talsma, D.
- Venue
- Frontiers in Integrative Neuroscience
- Publisher
- Frontiers Media SA
- Published
- 2015-03-26
- Volume
- 09
- Pages
- 19-19
- ISSN
- 1662-5145 (ISSN-L)
- OpenAlex
- W1982657229
- Cited by
- 222
- References
- 29
- FWCI
- 26.86
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.