Computational principles and models of multisensory integration
Open accessThe DOI matches its Crossref record; 25 of the 25 references it cites were found in a registry or an index.
Abstract
Combining information from multiple senses creates robust percepts, speeds up responses, enhances learning, and improves detection, discrimination, and recognition. In this review, I discuss computational models and principles that provide insight into how this process of multisensory integration occurs at the behavioral and neural level. My initial focus is on drift-diffusion and Bayesian models that can predict behavior in multisensory contexts. I then highlight how recent neurophysiological and perturbation experiments provide evidence for a distributed redundant network for multisensory integration. I also emphasize studies which show that task-relevant variables in multisensory contexts are distributed in heterogeneous neural populations. Finally, I describe dimensionality reduction methods and recurrent neural network models that may help decipher heterogeneous neural populations involved in multisensory integration.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Humans integrate visual and haptic information in a statistically optimal fashion
Ernst, M. O., & Banks, M. S. (2002) · Nature · doi:10.1038/415429a · Cited by 4,844
The Bayesian brain: The role of uncertainty in neural coding and computation
Knill, D. C., & Pouget, A. (2004) · Trends in Neurosciences · doi:10.1016/j.tins.2004.10.007 · Cited by 2,588 · cited 1×
“Bayesian decision theory models near-optimal discrimination and other behaviors such as identifying the underlying cause of events in the world [29,30**,31] as a statistical inference process performed by the brain [32] (Fig.”
BackgroundVisual Contribution to Speech Intelligibility in Noise
Sumby, W. H., & Pollack, I. (1954) · The Journal of the Acoustical Society of America · doi:10.1121/1.1907309 · Cited by 2,583
The Ventriloquist Effect Results from Near-Optimal Bimodal Integration
Alais, D., & Burr, D. (2004) · Current Biology · doi:10.1016/j.cub.2004.01.029 · Cited by 1,936
Merging the senses into a robust percept
Ernst, M. O., & Bülthoff, H. H. (2004) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2004.02.002 · Cited by 1,906 · cited 4×
“At every moment, our brains are combining cues from multiple senses to form robust percepts of signals in the world, a process termed multisensory integration [1].”
BackgroundInfluentialThe importance of mixed selectivity in complex cognitive tasks
Rigotti, M., Barak, O., Warden, M. R., Wang, X.-J., Daw, N. D., Miller, E. K., & Fusi, S. (2013) · Nature · doi:10.1038/nature12160 · Cited by 1,639 · cited 1×
“The consensus from several experiments is that single neuron responses in both uni-and multisensory brain regions are often perplexing, temporally complex and can be counterintuitive to expectations [75,77].”
BackgroundRoitman, J. D., & Shadlen, M. N. (2002) · The Journal of Neuroscience · doi:10.1523/jneurosci.22-21-09475.2002 · Cited by 1,589 · cited 1×
“Future neurophysiological studies using multisensory RT detection/discrimination paradigms will allow further quantitative tests of behavioral frameworks (such as the drift-diffusion and Bayesian models) and improve our understanding of this distributed network [67].”
Is neocortex essentially multisensory?
GHAZANFAR, A., & SCHROEDER, C. (2006) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2006.04.008 · Cited by 1,463
Bayesian inference with probabilistic population codes
Ma, W. J., Beck, J. M., Latham, P. E., & Pouget, A. (2006) · Nature Neuroscience · doi:10.1038/nn1790 · Cited by 1,457 · cited 1×
“The Bayesian framework’s success in explaining behavior has inspired the development of neural network models that can represent probability distributions [34] and perform near-optimal Bayesian inference as well as other important probabilistic computations [17,35].”
BackgroundDimensionality reduction for large-scale neural recordings
Cunningham, J. P., & Yu, B. M. (2014) · Nature Neuroscience · doi:10.1038/nn.3776 · Cited by 1,219 · cited 1×
“The goal of the approach is to understand the “dynamics” of this lower dimensional population neural “state,” (how it evolves through time) and the relationship of the dynamics to various features of the stimulus and the behavior [75,76,80,81,82*].”
BackgroundCortical oscillations and sensory predictions
Arnal, L. H., & Giraud, A.-L. (2012) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2012.05.003 · Cited by 1,114
Causal Inference in Multisensory Perception
Körding, K. P., Beierholm, U., Ma, W. J., Quartz, S., Tenenbaum, J. B., & Shams, L. (2007) · PLoS ONE · doi:10.1371/journal.pone.0000943 · Cited by 1,069 · cited 1×
“Bayesian decision theory models near-optimal discrimination and other behaviors such as identifying the underlying cause of events in the world [29,30**,31] as a statistical inference process performed by the brain [32] (Fig.”
BackgroundDIVISION OF PSYCHOLOGY : STATISTICAL FACILITATION OF SIMPLE REACTION TIMES*
Raab, D. H. (1962) · Transactions of the New York Academy of Sciences · doi:10.1111/j.2164-0947.1962.tb01433.x · Cited by 668
Probabilistic brains: knowns and unknowns
Pouget, A., Beck, J. M., Ma, W. J., & Latham, P. E. (2013) · Nature Neuroscience · doi:10.1038/nn.3495 · Cited by 615 · cited 3×
“Finally, combining the dynamical features of these RNNs with principles of probabilistic computation [35]”
BackgroundMethodMultisensory Flavor Perception
Spence, C. (2015) · Cell · doi:10.1016/j.cell.2015.03.007 · Cited by 579 · cited 1×
“For instance, the percept of flavor involves integration of gustatory and olfactory cues [2,3].”
BackgroundNeural correlates of multisensory cue integration in macaque MSTd
Gu, Y., Angelaki, D. E., & DeAngelis, G. C. (2008) · Nature Neuroscience · doi:10.1038/nn.2191 · Cited by 554 · cited 4×
“Monkeys, when trained to report their heading direction, also near-optimally combine visual (optic flow), and vestibular cues [27].”
BackgroundInfluentialA multilevel multimodal circuit enhances action selection in drosophila
Ohyama, T., Schneider-Mizell, C. M., Fetter, R. D., Aleman, J. V., Franconville, R., Rivera-Alba, M., Mensh, B. D., Branson, K. M., Simpson, J. H., Truman, J. W., et al. (2015) · Nature · doi:10.1038/nature14297 · Cited by 502 · cited 1×
“Finally, a groundbreaking study combining electron microscopy, electrophysiology, and behavior demonstrated that a multilevel multisensory circuit contributes to the escape response in larval Drosophila [65**].”
BackgroundNeural correlates of reliability-based cue weighting during multisensory integration
Fetsch, C. R., Pouget, A., DeAngelis, G. C., & Angelaki, D. E. (2012) · Nature Neuroscience · doi:10.1038/nn.2983 · Cited by 459 · cited 1×
“They are also sensitive to changes in the reliability of these cues on a trial-by-trial basis [49].”
BackgroundMultisensory integration: psychophysics, neurophysiology, and computation
Angelaki, D. E., Gu, Y., & DeAngelis, G. C. (2009) · Current Opinion in Neurobiology · doi:10.1016/j.conb.2009.06.008 · Cited by 383 · cited 1×
“However, the effortless nature of multisensory integration hides the underlying complexity of this process [11–17].”
BackgroundCortical Hierarchies Perform Bayesian Causal Inference in Multisensory Perception
Rohe, T., & Noppeney, U. (2015) · PLOS Biology · doi:10.1371/journal.pbio.1002073 · Cited by 344 · cited 1×
“Bayesian decision theory models near-optimal discrimination and other behaviors such as identifying the underlying cause of events in the world [29,30**,31] as a statistical inference process performed by the brain [32] (Fig.”
BackgroundEvaluating the Operations Underlying Multisensory Integration in the Cat Superior Colliculus
Stanford, T. R., Quessy, S., & Stein, B. E. (2005) · The Journal of Neuroscience · doi:10.1523/jneurosci.5095-04.2005 · Cited by 314
Neural Coding of Uncertainty and Probability
Ma, W. J., & Jazayeri, M. (2014) · Annual Review of Neuroscience · doi:10.1146/annurev-neuro-071013-014017 · Cited by 290 · cited 1×
“Bayesian models assume that probability distributions of prior knowledge and sensory information (likelihood) are known [1,24,33].”
BackgroundSong, H. F., Yang, G. R., & Wang, X.-J. (2016) · PLOS Computational Biology · doi:10.1371/journal.pcbi.1004792 · Cited by 281
Neural circuits as computational dynamical systems
Sussillo, D. (2014) · Current Opinion in Neurobiology · doi:10.1016/j.conb.2014.01.008 · Cited by 247
Learning Multisensory Integration and Coordinate Transformation via Density Estimation
Makin, J. G., Fellows, M. R., & Sabes, P. N. (2013) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1003035 · Cited by 53 · cited 2×
“Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain. delayed relative to each other [5,8,18], and exist in different reference frames [9,19].”
Background
Cited by
Learning and attention reveal a general relationship between population activity and behavior
Ni, A. M., Ruff, D. A., Alberts, J. J., Symmonds, J., & Cohen, M. R. (2018) · Science · doi:10.1126/science.aao0284 · Cited by 188
Cognition as a Window into Neuronal Population Space
Ruff, D. A., Ni, A. M., & Cohen, M. R. (2018) · Annual Review of Neuroscience · doi:10.1146/annurev-neuro-080317-061936 · Cited by 63
“Provocatively, many of these studies suggesting optimal decoding strategies have used multisensory stimuli or otherwise involved the integration of multiple modalities of sensory information (Chandrasekaran 2017, Kording & Wolpert 2006).”
BackgroundAuditory information enhances post-sensory visual evidence during rapid multisensory decision-making
Franzen, L., Delis, I., De Sousa, G., Kayser, C., & Philiastides, M. G. (2020) · Nature Communications · doi:10.1038/s41467-020-19306-7 · Cited by 57
Tech-Art-Theory: Improvisational Methods for HCI Learning and Teaching
Kang, L., & Jackson, S. (2021) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3449156 · Cited by 14
Metadata
- Authors
- Chandrasekaran, C.
- Venue
- Current Opinion in Neurobiology
- Publisher
- Elsevier BV
- Published
- 2017-04
- Volume
- 43
- Pages
- 25-34
- ISSN
- 0959-4388 (ISSN-L)
- OpenAlex
- W2558579624
- Cited by
- 103
- References
- 25
- FWCI
- 1.33
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.