A Bayesian View on Multimodal Cue Integration
Open accessAbstract
We perceive our own body and the world surrounding us via multiple sources of sensory information derived from several modalities, including vision, touch, and audition. To enable interactions with the environment, this information has to converge into a coherent and unambiguous multimodal percept of the body and the world. But how does the brain come up with such a unique percept? In this chapter, I review a model that in the statistical sense describes an optimal integration mechanism. The benefit of integrating sensory information comes from a reduction in variance of the final perceptual estimate. Furthermore, I point out how this integration scheme can be incorporated into a larger framework using Bayesian decision theory (BDT).
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Adults' integration of sensory information comes at a cost of losing access to the individual estimates that feed into the integrated percept ("sensory fusion").
“In adults, sensory integration can lead to mandatory “fusion,” in which the ability to judge the individual component estimates is lost (12, 13).”
From Fusion of visual cues is not mandatory in children (2010) · cites this paper as [13]
Fusion entails a poor ability to detect sensory conflicts.
“Fusion entails a poor ability to detect sensory conflicts (13).”
From Fusion of visual cues is not mandatory in children (2010) · cites this paper as [13]
Cites
(1966) · Medical Entomology and Zoology · Cited by 10,293 · see citing works
Humans integrate visual and haptic information in a statistically optimal fashion
Ernst, M. O., & Banks, M. S. (2002) · Nature · doi:10.1038/415429a · Cited by 4,844 · see citing works
Self-citationBayesian integration in sensorimotor learning
Körding, K. P., & Wolpert, D. M. (2004) · Nature · doi:10.1038/nature02169 · Cited by 2,004 · see citing works
The Ventriloquist Effect Results from Near-Optimal Bimodal Integration
Alais, D., & Burr, D. (2004) · Current Biology · doi:10.1016/j.cub.2004.01.029 · Cited by 1,936 · see citing works
Merging the senses into a robust percept
Ernst, M. O., & Bülthoff, H. H. (2004) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2004.02.002 · Cited by 1,906 · see citing works
Self-citationPerception as Bayesian Inference
(1996) · doi:10.1017/cbo9780511984037 · Cited by 1,425 · see citing works
Object perception as Bayesian inference
Kersten, D., Mamassian, P., & Yuille, A. (2004) · Annual Review of Psychology · doi:10.1146/annurev.psych.55.090902.142005 · Cited by 1,269 · see citing works
Motion illusions as optimal percepts
Weiss, Y., Simoncelli, E. P., & Adelson, E. H. (2002) · Nature Neuroscience · doi:10.1038/nn0602-858 · Cited by 1,076 · see citing works
Measurement and modeling of depth cue combination: in defense of weak fusion
Landy, M. S., Maloney, L. T., Johnston, E. B., & Young, M. (1995) · Vision Research · doi:10.1016/0042-6989(94)00176-m · Cited by 996 · see citing works
Graham, N. (1989) · Oxford University Press eBooks · doi:10.1093/acprof:oso/9780195051544.001.0001 · Cited by 959 · see citing works
Integration of Proprioceptive and Visual Position-Information: An Experimentally Supported Model
van Beers, R. J., Sittig, A. C., & Gon, J. J. D. V. D. (1999) · Journal of Neurophysiology · doi:10.1152/jn.1999.81.3.1355 · Cited by 742 · see citing works
Data Fusion for Sensory Information Processing Systems
Clark, J. J., & Yuille, A. (1990) · Springer eBooks · doi:10.1007/978-1-4757-2076-1 · Cited by 519 · see citing works
Combining Sensory Information: Mandatory Fusion Within, but Not Between, Senses
Hillis, J. M., Ernst, M. O., Banks, M. S., & Landy, M. S. (2002) · Science · doi:10.1126/science.1075396 · Cited by 486 · see citing works
Self-citationProblems Arising in the Analysis of a Series of Similar Experiments
Cochran, W. G. (1937) · Journal of the Royal Statistical Society Series B (Statistical Methodology) · doi:10.2307/2984123 · Cited by 461 · see citing works
Perceptual Modification: Adapting to Altered Sensory Environments
Welch, R. B. (1978) · Cited by 455 · see citing works
Slant from texture and disparity cues: Optimal cue combination
Hillis, J. M., Watt, S. J., Landy, M. S., & Banks, M. S. (2004) · Journal of Vision · doi:10.1167/4.12.1 · Cited by 421 · see citing works
Do humans optimally integrate stereo and texture information for judgments of surface slant?
Knill, D. C., & Saunders, J. A. (2003) · Vision Research · doi:10.1016/s0042-6989(03)00458-9 · Cited by 403 · see citing works
The precision of proprioceptive position sense
van Beers, R. J., Sittig, A. C., & van der Gon, J. J. D. (1998) · Experimental Brain Research · doi:10.1007/s002210050525 · Cited by 396 · see citing works
Experience can change the 'light-from-above' prior
Adams, W. J., Graf, E. W., & Ernst, M. O. (2004) · Nature Neuroscience · doi:10.1038/nn1312 · Cited by 377 · see citing works
Self-citationBayesian models of object perception
Kersten, D., & Yuille, A. (2003) · Current Opinion in Neurobiology · doi:10.1016/s0959-4388(03)00042-4 · Cited by 301 · see citing works
The ventriloquist effect does not depend on the direction of deliberate visual attention
Bertelson, P., Vroomen, J., de Gelder, B., & Driver, J. (2000) · Perception & Psychophysics · doi:10.3758/bf03205552 · Cited by 252 · see citing works
Weighted linear cue combination with possibly correlated error
Oruç, İ., Maloney, L. T., & Landy, M. S. (2003) · Vision Research · doi:10.1016/s0042-6989(03)00435-8 · Cited by 227 · see citing works
The ventriloquist effect does not depend on the direction of automatic visual attention
Vroomen, J., Bertelson, P., & de Gelder, B. (2001) · Perception & Psychophysics · doi:10.3758/bf03194427 · Cited by 196 · see citing works
Viewing Geometry Determines How Vision and Haptics Combine in Size Perception
Gepshtein, S., & Banks, M. S. (2003) · Current Biology · doi:10.1016/s0960-9822(03)00133-7 · Cited by 171 · see citing works
Feeling what you hear: auditory signals can modulate tactile tap perception
Bresciani, J., Ernst, M. O., Drewing, K., Bouyer, G., Maury, V., & Kheddar, A. (2004) · Experimental Brain Research · doi:10.1007/s00221-004-2128-2 · Cited by 168 · see citing works
Self-citationDiscrimination of planar surface slant from texture: human and ideal observers compared
Knill, D. C. (1998) · Vision Research · doi:10.1016/s0042-6989(97)00325-8 · Cited by 148 · see citing works
Ideal cue combination for localizing texture-defined edges
Landy, M. S., & Kojima, H. (2001) · Journal of the Optical Society of America A · doi:10.1364/josaa.18.002307 · Cited by 137 · see citing works
Perception of 3D surface orientation from skew symmetry
Saunders, J. A., & Knill, D. C. (2001) · Vision Research · doi:10.1016/s0042-6989(01)00187-0 · Cited by 87 · see citing works
High-level Vision as Statistical Inference
Kersten, D. (1999) · Cited by 80 · see citing works
How Optimal Depth Cue Integration Depends on the Task
Schrater, P., & Kersten, D. (2000) · International Journal of Computer Vision · doi:10.1023/a:1026557704054 · Cited by 47 · see citing works
Bayesian combination of ambiguous shape cues
Adams, W. J., & Mamassian, P. (2004) · Journal of Vision · doi:10.1167/4.10.7 · Cited by 34 · see citing works
Gepshtein, S., Burge, J., Banks, M. S., et al. (2004) · Journal of Vision · doi:10.1167/4.8.140 · Cited by 1 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Crossmodal correspondences: A tutorial review
Spence, C. (2011) · Attention, Perception, & Psychophysics · doi:10.3758/s13414-010-0073-7 · Cited by 1,618 · see citing works
Relevance of Error: What Drives Motor Adaptation?
Wei, K., & Körding, K. (2009) · Journal of Neurophysiology · doi:10.1152/jn.90545.2008 · Cited by 422 · see citing works
Development of Cue Integration in Human Navigation
Nardini, M., Jones, P., Bedford, R., & Braddick, O. (2008) · Current Biology · doi:10.1016/j.cub.2008.04.021 · Cited by 431 · see citing works
The statistical determinants of adaptation rate in human reaching
Burge, J., Ernst, M. O., & Banks, M. S. (2008) · Journal of Vision · doi:10.1167/8.4.20 · Cited by 400 · see citing works
Self-citationLearning to integrate arbitrary signals from vision and touch
Ernst, M. O. (2007) · Journal of Vision · doi:10.1167/7.5.7 · Cited by 332 · see citing works
Self-citationParise, C. V., & Spence, C. (2009) · PLoS ONE · doi:10.1371/journal.pone.0005664 · Cited by 272 · see citing works
Fusion of visual cues is not mandatory in children
Nardini, M., Bedford, R., & Mareschal, D. (2010) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1001699107 · Cited by 161 · see citing works
Zhang, W.-H., Wang, H., Chen, A., Gu, Y., Lee, T. S., Wong, K. M., & Wu, S. (2019) · eLife · doi:10.7554/elife.43753 · Cited by 45 · see citing works
An Analysis of a Ring Attractor Model for Cue Integration
Sun, X., Mangan, M., & Yue, S. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-95972-6_49 · Cited by 27 · see citing works
- Authors
- Ernst, M. O.
- Affiliations
- Max Planck Society (DE)
- Corresponding author
- Ernst, Marc O
- ORCID
- Ernst, Marc O
- Venue
- Human Body Perception From The Inside Out
- Publisher
- Oxford University PressNew York, NY
- Published
- 2005-12-04
- Pages
- 105-131
- OpenAlex
- W1554090946
- Cited by
- 209
- References
- 32
- Citations by year
- 2026: 8 · 2025: 13 · 2024: 10 · 2023: 6 · 2022: 8 · 2021: 16 · 2020: 7 · 2019: 11 · 2018: 19 · 2017: 11 · 2016: 12 · 2015: 17 · 2014: 8 · 2013: 9 · 2012: 7
- FWCI
- 17.50
- Type
- book chapter
- Language
- en
- Open access
- green
- hdl.handle.net
- Licence
- other-oa
- Locations
- PUB – Publications at Bielefeld University (Bielefeld University) (submittedVersion)
- Publikationen an der Universität Bielefeld (Universität Bielefeld) (submittedVersion)
- MPG.PuRe (Max Planck Society) (submittedVersion, other-oa)
- Topics
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Multisensory perception and integration
- Life Sciences › Neuroscience › Cognitive Neuroscience › Tactile and Sensory Interactions
- Life Sciences › Neuroscience › Cognitive Neuroscience › Visual perception and processing mechanisms
- Keywords
- multimodal cue integration, Bayesian decision theory, perceptual integration, sensory input, variance reduction, perceptual estimation, multisensory perception, optimal integration
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
