Computational principles and models of multisensory integration
Open accessThe DOI matches its Crossref record; 25 of the 25 references it cites were found in a registry or an index.
Abstract
Combining information from multiple senses creates robust percepts, speeds up responses, enhances learning, and improves detection, discrimination, and recognition. In this review, I discuss computational models and principles that provide insight into how this process of multisensory integration occurs at the behavioral and neural level. My initial focus is on drift-diffusion and Bayesian models that can predict behavior in multisensory contexts. I then highlight how recent neurophysiological and perturbation experiments provide evidence for a distributed redundant network for multisensory integration. I also emphasize studies which show that task-relevant variables in multisensory contexts are distributed in heterogeneous neural populations. Finally, I describe dimensionality reduction methods and recurrent neural network models that may help decipher heterogeneous neural populations involved in multisensory integration.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Humans integrate visual and haptic information in a statistically optimal fashion
Ernst, M. O., & Banks, M. S. (2002) · Nature · doi:10.1038/415429a · Cited by 4,844
The Bayesian brain: The role of uncertainty in neural coding and computation
Knill, D. C., & Pouget, A. (2004) · Trends in Neurosciences · doi:10.1016/j.tins.2004.10.007 · Cited by 2,588 · cited 1×
“Bayesian decision theory models near-optimal discrimination and other behaviors such as identifying the underlying cause of events in the world [29,30**,31] as a statistical inference process performed by the brain [32] (Fig.”
BackgroundVisual Contribution to Speech Intelligibility in Noise
Sumby, W. H., & Pollack, I. (1954) · The Journal of the Acoustical Society of America · doi:10.1121/1.1907309 · Cited by 2,583
The Ventriloquist Effect Results from Near-Optimal Bimodal Integration
Alais, D., & Burr, D. (2004) · Current Biology · doi:10.1016/j.cub.2004.01.029 · Cited by 1,936
Merging the senses into a robust percept
Ernst, M. O., & Bülthoff, H. H. (2004) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2004.02.002 · Cited by 1,906 · cited 4×
“At every moment, our brains are combining cues from multiple senses to form robust percepts of signals in the world, a process termed multisensory integration [1].”
BackgroundInfluentialThe importance of mixed selectivity in complex cognitive tasks
Rigotti, M., Barak, O., Warden, M. R., Wang, X.-J., Daw, N. D., Miller, E. K., & Fusi, S. (2013) · Nature · doi:10.1038/nature12160 · Cited by 1,639 · cited 1×
“The consensus from several experiments is that single neuron responses in both uni-and multisensory brain regions are often perplexing, temporally complex and can be counterintuitive to expectations [75,77].”
BackgroundRoitman, J. D., & Shadlen, M. N. (2002) · The Journal of Neuroscience · doi:10.1523/jneurosci.22-21-09475.2002 · Cited by 1,589 · cited 1×
“Future neurophysiological studies using multisensory RT detection/discrimination paradigms will allow further quantitative tests of behavioral frameworks (such as the drift-diffusion and Bayesian models) and improve our understanding of this distributed network [67].”
Is neocortex essentially multisensory?
GHAZANFAR, A., & SCHROEDER, C. (2006) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2006.04.008 · Cited by 1,463
Bayesian inference with probabilistic population codes
Ma, W. J., Beck, J. M., Latham, P. E., & Pouget, A. (2006) · Nature Neuroscience · doi:10.1038/nn1790 · Cited by 1,457 · cited 1×
“The Bayesian framework’s success in explaining behavior has inspired the development of neural network models that can represent probability distributions [34] and perform near-optimal Bayesian inference as well as other important probabilistic computations [17,35].”
BackgroundDimensionality reduction for large-scale neural recordings
Cunningham, J. P., & Yu, B. M. (2014) · Nature Neuroscience · doi:10.1038/nn.3776 · Cited by 1,219 · cited 1×
“The goal of the approach is to understand the “dynamics” of this lower dimensional population neural “state,” (how it evolves through time) and the relationship of the dynamics to various features of the stimulus and the behavior [75,76,80,81,82*].”
Background
Cited by
Learning and attention reveal a general relationship between population activity and behavior
Ni, A. M., Ruff, D. A., Alberts, J. J., Symmonds, J., & Cohen, M. R. (2018) · Science · doi:10.1126/science.aao0284 · Cited by 188
Cognition as a Window into Neuronal Population Space
Ruff, D. A., Ni, A. M., & Cohen, M. R. (2018) · Annual Review of Neuroscience · doi:10.1146/annurev-neuro-080317-061936 · Cited by 63
“Provocatively, many of these studies suggesting optimal decoding strategies have used multisensory stimuli or otherwise involved the integration of multiple modalities of sensory information (Chandrasekaran 2017, Kording & Wolpert 2006).”
BackgroundAuditory information enhances post-sensory visual evidence during rapid multisensory decision-making
Franzen, L., Delis, I., De Sousa, G., Kayser, C., & Philiastides, M. G. (2020) · Nature Communications · doi:10.1038/s41467-020-19306-7 · Cited by 57
Tech-Art-Theory: Improvisational Methods for HCI Learning and Teaching
Kang, L., & Jackson, S. (2021) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3449156 · Cited by 14
Metadata
- Authors
- Chandrasekaran, C.
- Venue
- Current Opinion in Neurobiology
- Publisher
- Elsevier BV
- Published
- 2017-04
- Volume
- 43
- Pages
- 25-34
- ISSN
- 0959-4388 (ISSN-L)
- OpenAlex
- W2558579624
- Cited by
- 103
- References
- 25
- FWCI
- 1.33
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.