The DOI matches its Crossref record; 72 of the 72 references it cites were found in a registry or an index.
Learning Multisensory Integration and Coordinate Transformation via Density Estimation
Open accessAbstract
Sensory processing in the brain includes three key operations: multisensory integration-the task of combining cues into a single estimate of a common underlying stimulus; coordinate transformations-the change of reference frame for a stimulus (e.g., retinotopic to body-centered) effected through knowledge about an intervening variable (e.g., gaze position); and the incorporation of prior information. Statistically optimal sensory processing requires that each of these operations maintains the correct posterior distribution over the stimulus. Elements of this optimality have been demonstrated in many behavioral contexts in humans and other animals, suggesting that the neural computations are indeed optimal. That the relationships between sensory modalities are complex and plastic further suggests that these computations are learned-but how? We provide a principled answer, by treating the acquisition of these mappings as a case of density estimation, a well-studied problem in machine learning and statistics, in which the distribution of observed data is modeled in terms of a set of fixed parameters and a set of latent variables. In our case, the observed data are unisensory-population activities, the fixed parameters are synaptic connections, and the latent variables are multisensory-population activities. In particular, we train a restricted Boltzmann machine with the biologically plausible contrastive-divergence rule to learn a range of neural computations not previously demonstrated under a single approach: optimal integration; encoding of priors; hierarchical integration of cues; learning when not to integrate; and coordinate transformation. The model makes testable predictions about the nature of multisensory representations.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures







![Figure 8 Tuning curves in the hidden layer. (A) Tuning curves for the multisensory-integration model/data (Figs. 1 and 2). The left column shows tuning curves in the space of joint angles for sixteen randomly chosen hidden units; the right column shows those same units for the arc of reach endpoints from [29]. The top row shows tuning curves for the trained model; the second row shows the same curves for the untrained model. The location of the arc in joint space is shown by the black slash through the tuning curves in the left column. Whereas the left-column tuning curves were collected for a single gain (), the right-column curves were collected for (blue), (green), and (red) (the same gain was used for both populations, vis and prop). (B) Example hidden-unit tuning curves from the coordinate transformation model for body-centered hand position (; see text for details), for two different gaze positions (red and green curves). The dashed blue curves show where the red tuning curves would lie for the second gaze position if they shifted completely with the eyes, as illustrated by the red arrows, i.e. if they were retinotopic. Some cells (second column) are; some are body-centered (first column); some partially shift (third column); and some even shift in the opposite direction of the gaze angle. (C) Coordinate-transforming cells can be tuned for any of the variables on the continuum from gaze angle (E), to retinotopic hand position (), to body-centered hand position (), to body-centered hand position plus gaze angle (). The histogram shows the distributions of such tunings in the hidden layer, using the analysis of [31].](https://figures.cochat.ai/paper-figures/95ce420e5a7b4290b2090f72d11fd0b3/learning-multisensory-integration-and-coordinate-transformation-via-density-figure-8.jpg)
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Normal processing in the brain is highly tolerant of abnormal patterns and classifies patterns according to best matching prior sensory experiences (19).”
From A neural interface provides long-term stable natural touch perception (2014) · cites this paper as [19]
Experimental — accuracy not yet measuredThe check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“The complex perceptions described in response to patterned stimulation intensity paradigms suggest that sensory perception is a pattern recognition activity (19).”
From A neural interface provides long-term stable natural touch perception (2014) · cites this paper as [19]
Experimental — accuracy not yet measuredIn the citing paper
“and related models (Makin et al., 2013, 2016) have been used to account for optimal multi-sensory integration in the brain.”
From Toward an integration of deep learning and neuroscience (2016) · cites this paper as Makin et al., 2013
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Reducing the dimensionality of data with neural networks
Hinton, G. E., & Salakhutdinov, R. R. (2006) · Science · doi:10.1126/science.1127647 · Cited by 20,928
(2005) · Statistics for Biology and Health · doi:10.1007/0-387-27255-0_9 · Cited by 20,133
McCullagh, P., & Nelder, J. A. (1989) · doi:10.1007/978-1-4899-3242-6 · Cited by 16,836
A fast learning algorithm for deep belief nets
Hinton, G. E., Osindero, S., & Teh, Y.-W. (2006) · Neural Computation · doi:10.1162/neco.2006.18.7.1527 · Cited by 16,336 · cited 2×
“In particular, we model the cortical association area with a restricted Boltzmann machine (RBM), an undirected generative model trained with a fast, effective Hebbian learning rule, contrastive divergence [16,17].”
MethodTraining Products of Experts by Minimizing Contrastive Divergence
Hinton, G. E. (2002) · Neural Computation · doi:10.1162/089976602760128018 · Cited by 4,879 · cited 4×
“This rule approximates gradient descent on an objective function for density estimation (Hinton’s ‘‘contrastive divergence’’ [17], or alternatively ‘‘probability flow’’ [64]).”
BackgroundMethodInfluentialHumans integrate visual and haptic information in a statistically optimal fashion
Ernst, M. O., & Banks, M. S. (2002) · Nature · doi:10.1038/415429a · Cited by 4,844 · cited 2×
“based on the peak of the optimal posterior over the stimulus, given a variety of multisensory inputs [1,3–7].”
BackgroundSparse coding with an overcomplete basis set: A strategy employed by V1?
Olshausen, B. A., & Field, D. J. (1997) · Vision Research · doi:10.1016/s0042-6989(97)00169-7 · Cited by 3,642 · cited 1×
“One promising theoretical strategy would be to take ‘‘passing on all the information’’ as a given, and then to seek the set of constraints—fewest spikes [49], topography [50], fewest neurons, least processing time, computational efficiency [51], etc.”
BackgroundSome informational aspects of visual perception.
Attneave, F. (1954) · Psychological Review · doi:10.1037/h0054663 · Cited by 2,890 · cited 1×
“Finally, many authors have either anticipated [51,56,57] or”
BackgroundThe Bayesian brain: The role of uncertainty in neural coding and computation
Knill, D. C., & Pouget, A. (2004) · Trends in Neurosciences · doi:10.1016/j.tins.2004.10.007 · Cited by 2,588 · cited 1×
“based on the peak of the optimal posterior over the stimulus, given a variety of multisensory inputs [1,3–7].”
BackgroundBayesian integration in sensorimotor learning
Körding, K. P., & Wolpert, D. M. (2004) · Nature · doi:10.1038/nature02169 · Cited by 2,004
Cited by
A neural interface provides long-term stable natural touch perception
Tan, D. W., Schiefer, M. A., Keith, M. W., Anderson, J. R., Tyler, J., & Tyler, D. J. (2014) · Science Translational Medicine · doi:10.1126/scitranslmed.3008669 · Cited by 751
“Normal processing in the brain is highly tolerant of abnormal patterns and classifies patterns according to best matching prior sensory experiences (19).”
BackgroundBehavioral, Neural, and Computational Principles of Bodily Self-Consciousness
Blanke, O., Slater, M., & Serino, A. (2015) · Neuron · doi:10.1016/j.neuron.2015.09.029 · Cited by 698
Serino, A. (2019) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2019.01.016 · Cited by 304
“The neural mechanism allowing reference frame transformation system and the PPS multisensory integration system might interact, for instance, via intermediate neuronal layers between the unimodal and the multisensory layers, which re-code unisensory inputs into a common, body-part centered reference frame, necessary f…”
Grivaz, P., Blanke, O., & Serino, A. (2017) · NeuroImage · doi:10.1016/j.neuroimage.2016.12.052 · Cited by 203
“…during the rubber hand illusion is the so-called proprioceptive drift, whereby participants feel their hand to be closer to the location where they see the rubber hand during synchronous visuo-tactile stimulation (see also Makin et al., 2013; Samad et al., 2015 for computational approaches).”
A learning-based approach to artificial sensory feedback leads to optimal integration
Dadarlat, M. C., O'Doherty, J. E., & Sabes, P. N. (2015) · Nature Neuroscience · doi:10.1038/nn.3883 · Cited by 210
Self-citation
Metadata
- Authors
- Makin, J. G., Fellows, M. R., & Sabes, P. N.
- Affiliations
- University of California, San Francisco (US)
- Corresponding author
- Makin, Joseph G.
- Venue
- PLoS Computational Biology
- Publisher
- Public Library of Science (PLoS)
- Published
- 2013-04-18
- Volume
- 9
- Issue
- 4
- Pages
- e1003035
- ISSN
- 1553-734X (ISSN-L)
- 1553-7358
- PMC
- PMC3630212
- OpenAlex
- W2119669612
- Cited by
- 53
- References
- 71
- Citations by year
- 2026: 1 · 2025: 4 · 2024: 2 · 2023: 2 · 2022: 2 · 2021: 4 · 2020: 2 · 2019: 3 · 2017: 5 · 2016: 8 · 2015: 7 · 2014: 12 · 2013: 1
- FWCI
- 12.62
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Locations
- PLoS Computational Biology (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- eScholarship (California Digital Library) (submittedVersion)
- Figshare (submittedVersion, cc-by)
- Funders
- National Institutes of Health
- National Eye Institute (EY015679, R01EY015679)
- Topics
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Multisensory perception and integration
- Life Sciences › Neuroscience › Cognitive Neuroscience › Neural dynamics and brain function
- Life Sciences › Neuroscience › Cognitive Neuroscience › Visual perception and processing mechanisms
- Keywords
- multisensory integration, coordinate transformation, density estimation, restricted Boltzmann machine, contrastive divergence, prior information, hierarchical integration, population activity, gaze position
- MeSH terms
- Algorithms, Animals, Brain, Cues, Humans, Learning, Models, Neurological, Photic Stimulation, Probability, Sensation, Visual Perception, Normal Distribution, Poisson Distribution
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Learning Multisensory Integration and Coordinate Transformation via Density Estimation. https://app.cochat.ai/papers/10.1371/journal.pcbi.1003035
- The paper
- Makin, J. G., Fellows, M. R., & Sabes, P. N. (2013). Learning Multisensory Integration and Coordinate Transformation via Density Estimation. PLoS Computational Biology. https://doi.org/10.1371/journal.pcbi.1003035
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.