Humans integrate visual and haptic information in a statistically optimal fashion
The DOI matches its Crossref record; 25 of the 25 references it cites were found in a registry or an index.
Abstract
When a person looks at an object while exploring it with their hand, vision and touch both provide information for estimating the properties of the object. Vision frequently dominates the integrated visual-haptic percept, for example when judging size, shape or position, but in some circumstances the percept is clearly affected by haptics. Here we propose that a general principle, which minimizes variance in the final estimate, determines the degree to which vision or haptics dominates. This principle is realized by using maximum-likelihood estimation to combine the inputs. To investigate cue combination quantitatively, we first measured the variances associated with visual and haptic estimation of height. We then used these measurements to construct a maximum-likelihood integrator. This model behaved very similarly to humans in a visual-haptic task. Thus, the nervous system seems to combine visual and haptic information in a fashion that is similar to a maximum-likelihood integrator. Visual dominance occurs when the variance associated with visual estimation is lower than that associated with haptic estimation.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Multimodal integration is believed to follow a general principle; thus, the nervous system weights the information available in each modality in an optimal way (reducing final estimation variance), whereas attention can influence these weights (Alais & Burr, 2004;Ernst & Banks, 2002;v a nB e e r se ta l .,1999).”
From Augmented visual, auditory, haptic, and multimodal feedback in motor learning: A review (2013) · cites this paper as Ernst & Banks, 2002
In the citing paper
“Interestingly, human observers often approach such an 'ideal observer' ([102], but see [103]), inspiring the idea that the human perceptual system may implement (approximate) Bayesian inference [104].”
From How Do Expectations Shape Perception? (2018) · cites this paper as [102]
In the citing paper
“Moreover, the relative weighting of different information sources according to their reliability in this process is suggestive of the deployment of precision expectations which underlie optimal cue combination in a Bayesian scheme [51].”
From Interoceptive inference, emotion, and the embodied self (2013) · cites this paper as [51]
In the citing paper
“In cue integration paradigms, such as the classic visual–haptic study by Ernst and Banks (2002), retrospective model-fitting often justifies Bayesian optimality post hoc, as priors and likelihoods are typically inferred from the behavioral data they purport to explain, rather than derived independently.”
From The myth of the Bayesian brain (2025) · cites this paper as Ernst and Banks 2002
In the citing paper
“Accurate inference requires that more informative cues should be weighted more heavily and, in some cases, this reweighting process can occur instantaneously; for example, we immediately ascribe less weight to a visual cue when it is blurred or degraded51,52.”
From Multimodal cue integration and learning in a neural representation of head direction (2025) · cites this paper as [52]
8 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Immediate perceptual response to intersensory discrepancy.
Welch, R. B., & Warren, D. H. (1980) · Psychological Bulletin · doi:10.1037/0033-2909.88.3.638 · Cited by 1,176 · cited 1×
“Shaded area represents predicted weights expected from within-modality discrimination (a; equation (5)); its height represents predicted errors given the standard errors of the within-modality discrimination.”
BackgroundMeasurement and modeling of depth cue combination: in defense of weak fusion
Landy, M. S., Maloney, L. T., Johnston, E. B., & Young, M. (1995) · Vision Research · doi:10.1016/0042-6989(94)00176-m · Cited by 997
Integration of Proprioceptive and Visual Position-Information: An Experimentally Supported Model
van Beers, R. J., Sittig, A. C., & Gon, J. J. D. V. D. (1999) · Journal of Neurophysiology · doi:10.1152/jn.1999.81.3.1355 · Cited by 742
Vision and Touch: An Experimentally Created Conflict between the Two Senses
Rock, I., & Victor, J. (1964) · Science · doi:10.1126/science.143.3606.594 · Cited by 702
Visual Capture of Touch: Out-of-the-Body Experiences With Rubber Gloves.
Pavani, F., Spence, C., & Driver, J. (2000) · Psychological Science · doi:10.1111/1467-9280.00270 · Cited by 608
Adaptation, after-effect and contrast in the perception of curved lines.
Gibson, J. J. (1933) · Journal of Experimental Psychology · doi:10.1037/h0074626 · Cited by 599
Data Fusion for Sensory Information Processing Systems
Clark, J. J., & Yuille, A. L. (1990) · Springer eBooks · doi:10.1007/978-1-4757-2076-1 · Cited by 525
Visual capture produced by prism spectacles
Hay, J. C., Pick, H. L., & Ikeda, K. (1965) · Psychonomic Science · doi:10.3758/bf03343413 · Cited by 309
Touch can change visual slant perception
Ernst, M. O., Banks, M. S., & Bülthoff, H. H. (2000) · Nature Neuroscience · doi:10.1038/71140 · Cited by 236
Self-citationAuditory Spatial Receptive Fields Created by Multiplication
Peña, J. L., & Konishi, M. (2001) · Science · doi:10.1126/science.1059201 · Cited by 222
Research works printed but not linked: 1
- Tastevin, J. En partant de l’experience d’Aristote. L’Encephale 1, 57–84 (1937).
Cited by
Items 1,001–1,044 of 1,044
Zhang, W.-H., Wang, H., Chen, A., Gu, Y., Lee, T. S., Wong, K. M., & Wu, S. (2019) · eLife · doi:10.7554/elife.43753 · Cited by 45
Auditory, tactile, and audiotactile information processing following visual deprivation
Occelli, V., Spence, C., & Zampini, M. (2013) · Psychological Bulletin · doi:10.1037/a0028416 · Cited by 43
Deane, G. (2021) · Neuroscience of Consciousness · doi:10.1093/nc/niab024 · Cited by 38
Artificial cognition vs. artificial intelligence for next-generation autonomous robotic agents
Sandini, G., Sciutti, A., & Morasso, P. (2024) · Frontiers in Computational Neuroscience · doi:10.3389/fncom.2024.1349408 · Cited by 40
“In Pre-natal and post-natal visuo-haptic experience. contrast, adults are able to optimally fuse multisensory information in an abstract form (Ernst and Banks, 2002).”
BackgroundPennartz, C. M. A. (2022) · Behavioural Brain Research · doi:10.1016/j.bbr.2022.113969 · Cited by 40
“ability to integrate sensory evidence on a stimulus and its properties from multiple sensory sources, relying on the notion of Bayesian inference [21-24].”
BackgroundWhole brain mapping of visual and tactile convergence in the macaque monkey
Guipponi, O., Cléry, J., Odouard, S., Wardak, C., & Ben Hamed, S. (2015) · NeuroImage · doi:10.1016/j.neuroimage.2015.05.022 · Cited by 40
Peri-personal Space as a Prior in Coupling Visual and Proprioceptive Signals
Noel, J.-P., Samad, M., Doxon, A., Clark, J., Keller, S., & Di Luca, M. (2018) · Scientific Reports · doi:10.1038/s41598-018-33961-3 · Cited by 39
Is that what Bayesians believe? Reply to Griffiths, Chater, Norris, and Pouget (2012).
Bowers, J. S., & Davis, C. J. (2012) · Psychological Bulletin · doi:10.1037/a0027750 · Cited by 38
(2011) · Cited by 38
Joint Representation of Depth from Motion Parallax and Binocular Disparity Cues in Macaque Area MT
Nadler, J. W., Barbash, D., Kim, H. R., Shimpi, S., Angelaki, D. E., & DeAngelis, G. C. (2013) · Journal of Neuroscience · doi:10.1523/jneurosci.0251-13.2013 · Cited by 37
Bayesian inference in ring attractor networks
Kutschireiter, A., Basnak, M. A., Wilson, R. I., & Drugowitsch, J. (2023) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2210622120 · Cited by 36
Wei, K., Stevenson, I. H., & Kording, K. P. (2010) · Journal of Vision · doi:10.1167/10.14.4 · Cited by 36
“…of evidence indicating that the nervous system combines sensory cues near optimally for estimation and for movement control (Alais & Burr, 2004; Ernst & Banks, 2002; Ernst & Bülthoff, 2004; Knill & Pouget, 2004; Körding & Wolpert, 2004; Korenberg & Ghahramani, 2002; van Beers, 2009; van Beers,…”
Coello, Y., Quesque, F., Gigliotti, M.-F., Ott, L., & Bruyelle, J.-L. (2018) · PLOS ONE · doi:10.1371/journal.pone.0196874 · Cited by 35
Kim, H. R., Pitkow, X., Angelaki, D. E., & DeAngelis, G. C. (2016) · Journal of Neurophysiology · doi:10.1152/jn.00005.2016 · Cited by 32
Chen, W., Chao, J.-G., Chen, X.-W., Wang, J.-K., & Tan, C. (2015) · Brain Research Bulletin · doi:10.1016/j.brainresbull.2015.01.007 · Cited by 29
Proscription supports robust perceptual integration by suppression in human visual cortex
Rideaux, R., & Welchman, A. E. (2018) · Nature Communications · doi:10.1038/s41467-018-03400-y · Cited by 27
An Analysis of a Ring Attractor Model for Cue Integration
Sun, X., Mangan, M., & Yue, S. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-95972-6_49 · Cited by 27
Chung, W., & Barnett-Cowan, M. (2023) · Virtual Reality · doi:10.1007/s10055-023-00786-z · Cited by 26
Sensory augmentation: integration of an auditory compass signal into human perception of space
Schumann, F., & O’Regan, J. K. (2017) · Scientific Reports · doi:10.1038/srep42197 · Cited by 25
Tseng, Y.-T., Tsai, C.-L., Chen, F.-C., & Konczak, J. (2019) · Journal of Motor Behavior · doi:10.1080/00222895.2017.1415200 · Cited by 24
“These gains may change following a minimum variance approach to achieve perceptual stability (Ernst & Banks, 2002).”
BackgroundPrior probability modulates anticipatory activity in category-specific areas
Trapp, S., Lepsien, J., Kotz, S. A., & Bar, M. (2016) · Cognitive, Affective, & Behavioral Neuroscience · doi:10.3758/s13415-015-0373-4 · Cited by 24
Detect, Reject, Correct: Crossmodal Compensation of Corrupted Sensors
Lee, M. A., Tan, M., Zhu, Y., & Bohg, J. (2021) · 2021 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra48506.2021.9561847 · Cited by 24
Suboptimal human multisensory cue combination
Arnold, D. H., Petrie, K., Murray, C., & Johnston, A. (2019) · Scientific Reports · doi:10.1038/s41598-018-37888-7 · Cited by 23
“These data are generally consistent with previous reports of cross-modal encoding advantages 5,12,13 , and specifically with bimodal audio-visual improvements in sensitivity to spatial position 4 and temporal modulation rates 14,15 .”
BackgroundMethodResultInfluentialTesting predictive coding theories of autism spectrum disorder using models of active inference
Arthur, T., Vine, S., Buckingham, G., Brosnan, M., Wilson, M., & Harris, D. (2023) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1011473 · Cited by 21
The macaque ventral intraparietal area has expanded into three homologue human parietal areas
Foster, C., Sheng, W.-A., Heed, T., & Ben Hamed, S. (2022) · Progress in Neurobiology · doi:10.1016/j.pneurobio.2021.102185 · Cited by 20
Predictive action perception from explicit intention information in autism
Hudson, M., Nicholson, T., Kharko, A., McKenzie, R., & Bach, P. (2021) · Psychonomic Bulletin & Review · doi:10.3758/s13423-021-01941-w · Cited by 19
“Instead, our findings imply a more differentiated account in which prediction channels for perceptually derived top-down information are down-weighted in ASD, while others remain intact, perhaps guided by the relative reliability of prediction sources (Ernst & Banks, 2002).”
BackgroundInfluentialNoel, J.-P., Simon, D., Thelen, A., Maier, A., Blake, R., & Wallace, M. T. (2018) · Journal of Cognitive Neuroscience · doi:10.1162/jocn_a_01247 · Cited by 17
“These models have been applied to psychophysical (Ernst & Banks, 2002), neuroimaging (Rohe & Noppeney, 2015, 2016), and electrophysiological (Boyle, Kayser, & Kayser, 2017; Fetsch, Deangelis, & Angelaki, 2013) observations concerning suprathreshold multisensory performance and collectively…”
Experimentally disambiguating models of sensory cue integration
Scarfe, P. (2022) · Journal of Vision · doi:10.1167/jov.22.1.5 · Cited by 17
Computational Models of Hallucinations
Jardri, R., & Denève, S. (2013) · The Neuroscience of Hallucinations · doi:10.1007/978-1-4614-4121-2_16 · Cited by 16
Multisensory integration and behavioral stability
Roy, C., Dalla Bella, S., Pla, S., & Lagarde, J. (2021) · Psychological Research · doi:10.1007/s00426-019-01273-4 · Cited by 13
Human inference in changing environments with temporal structure.
Prat-Carrabin, A., Wilson, R. C., Cohen, J. D., & Azeredo da Silveira, R. (2021) · Psychological Review · doi:10.1037/rev0000276 · Cited by 16
“In cognitive science, it has successfully accounted for a wide range of observations, including cue combination in humans [5, 4, 1, 61, 3, 6, 7, 2], sensorimotor control [8, 9, 62], integration of temporal statistics [37, 36, 10, 11, 38], perceptual multistability [63, 41, 42], and various aspects of cognition [64, 65…”
BackgroundMultimodal cue integration and learning in a neural representation of head direction
Basnak, M. A., Kutschireiter, A., Okubo, T. S., Chen, A., Gorelik, P., Drugowitsch, J., & Wilson, R. I. (2025) · Nature Neuroscience · doi:10.1038/s41593-024-01823-z · Cited by 13
The myth of the Bayesian brain
Mangalam, M. (2025) · European Journal of Applied Physiology · doi:10.1007/s00421-025-05855-6 · Cited by 14
“A classic example comes from studies on visual-haptic integration, which report that humans combine sensory cues in a statistically optimal way, as predicted by Bayesian models (Ernst and Banks 2002).”
Eye Movements: Illusions in Slow Motion
Ernst, M. O. (2010) · Current Biology · doi:10.1016/j.cub.2010.03.009 · Cited by 10
Self-citationWeighted cue integration for straight-line orientation
Shaverdian, S., Dirlik, E., Mitchell, R., Tocco, C., Webb, B., & Dacke, M. (2022) · iScience · doi:10.1016/j.isci.2022.105207 · Cited by 9
Scherer, J., Müller, M. M., Unterbrink, P., Meier, S., Egelhaaf, M., Bertrand, O. J. N., & Boeddeker, N. (2024) · Frontiers in Behavioral Neuroscience · doi:10.3389/fnbeh.2024.1399716 · Cited by 8
Kumar, N., & Mutha, P. K. (2016) · Journal of Neurophysiology · doi:10.1152/jn.00850.2015 · Cited by 7
Tottori, T., & Kobayashi, T. J. (2025) · Physical Review Research · doi:10.1103/gvl6-cvby · Cited by 6
Multisensory Integration in Body Representation
Fang, W., Liu, Y., & Wang, L. (2024) · Advances in Experimental Medicine and Biology · doi:10.1007/978-981-99-7611-9_5 · Cited by 5
Resource limitations induce phase transitions in biological information processing
Tottori, T., & Kobayashi, T. J. (2025) · Physical Review Research · doi:10.1103/nz4j-tyv1 · Cited by 3
(2020) · doi:10.1017/9781108339216 · Cited by 2
The generalized Hierarchical Gaussian Filter
Weber, L. A., Waade, P. T., Legrand, N., Møller, A. H., Stephan, K. E., & Mathys, C. (2026) · eLife · doi:10.7554/elife.110174
Kuo, P.-C., Hou, H., Dabney, W., & Walker, E. Y. (2025) · arXiv
“Humans and animals are shown to achieve near Bayes-optimal behavior under uncertainty, leading to theories and experiments investigating efficient Bayes-optimal representation and computation in the brain [14, 15, 16].”
The generalized Hierarchical Gaussian Filter
Weber, L. A., Waade, P. T., Legrand, N., Møller, A. H., Stephan, K. E., & Mathys, C. (2026) · doi:10.7554/elife.110174.1
Metadata
- Authors
- Ernst, M. O., & Banks, M. S.
- Affiliations
- Max Planck Institute for Biological Cybernetics (DE)
- University of California, Berkeley (US)
- Venue
- Nature
- Published
- 2002-01
- Volume
- 415
- Issue
- 6870
- Pages
- 429-433
- ISSN
- 0028-0836 (ISSN-L)
- 1476-4687
- OpenAlex
- W1969090956
- Cited by
- 4,844
- References
- 25
- Citations by year
- 2026: 237 · 2025: 267 · 2024: 260 · 2023: 287 · 2022: 220 · 2021: 279 · 2020: 311 · 2019: 316 · 2018: 267 · 2017: 275 · 2016: 192 · 2015: 230 · 2014: 207 · 2013: 195 · 2012: 198
- FWCI
- 80.64
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- www.springer.com/tdm (tdm)
- Locations
- Nature (publishedVersion)
- PubMed (publishedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- Max Planck Institute for Plasma Physics (submittedVersion)
- PUB – Publications at Bielefeld University (Bielefeld University) (submittedVersion)
- Publikationen an der Universität Bielefeld (Universität Bielefeld) (submittedVersion)
- MPG.PuRe (Max Planck Society) (submittedVersion)
- Funders
- National Science Foundation
- National Institutes of Health
- Topics
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Multisensory perception and integration
- Life Sciences › Neuroscience › Cognitive Neuroscience › Tactile and Sensory Interactions
- Life Sciences › Neuroscience › Cognitive Neuroscience › Visual perception and processing mechanisms
- Keywords
- visual-haptic integration, cue combination, maximum likelihood estimation, statistical optimality, visual dominance, visual estimation, height perception, variance minimization, multisensory perception
- MeSH terms
- Adult, Humans, Models, Neurological, Sensory Thresholds, Size Perception, Touch
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.