Humans integrate visual and haptic information in a statistically optimal fashion
The DOI matches its Crossref record; 25 of the 25 references it cites were found in a registry or an index.
Abstract
When a person looks at an object while exploring it with their hand, vision and touch both provide information for estimating the properties of the object. Vision frequently dominates the integrated visual-haptic percept, for example when judging size, shape or position, but in some circumstances the percept is clearly affected by haptics. Here we propose that a general principle, which minimizes variance in the final estimate, determines the degree to which vision or haptics dominates. This principle is realized by using maximum-likelihood estimation to combine the inputs. To investigate cue combination quantitatively, we first measured the variances associated with visual and haptic estimation of height. We then used these measurements to construct a maximum-likelihood integrator. This model behaved very similarly to humans in a visual-haptic task. Thus, the nervous system seems to combine visual and haptic information in a fashion that is similar to a maximum-likelihood integrator. Visual dominance occurs when the variance associated with visual estimation is lower than that associated with haptic estimation.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Multimodal integration is believed to follow a general principle; thus, the nervous system weights the information available in each modality in an optimal way (reducing final estimation variance), whereas attention can influence these weights (Alais & Burr, 2004;Ernst & Banks, 2002;v a nB e e r se ta l .,1999).”
From Augmented visual, auditory, haptic, and multimodal feedback in motor learning: A review (2013) · cites this paper as Ernst & Banks, 2002
In the citing paper
“Interestingly, human observers often approach such an 'ideal observer' ([102], but see [103]), inspiring the idea that the human perceptual system may implement (approximate) Bayesian inference [104].”
From How Do Expectations Shape Perception? (2018) · cites this paper as [102]
In the citing paper
“Moreover, the relative weighting of different information sources according to their reliability in this process is suggestive of the deployment of precision expectations which underlie optimal cue combination in a Bayesian scheme [51].”
From Interoceptive inference, emotion, and the embodied self (2013) · cites this paper as [51]
In the citing paper
“In cue integration paradigms, such as the classic visual–haptic study by Ernst and Banks (2002), retrospective model-fitting often justifies Bayesian optimality post hoc, as priors and likelihoods are typically inferred from the behavioral data they purport to explain, rather than derived independently.”
From The myth of the Bayesian brain (2025) · cites this paper as Ernst and Banks 2002
In the citing paper
“Accurate inference requires that more informative cues should be weighted more heavily and, in some cases, this reweighting process can occur instantaneously; for example, we immediately ascribe less weight to a visual cue when it is blurred or degraded51,52.”
From Multimodal cue integration and learning in a neural representation of head direction (2025) · cites this paper as [52]
8 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Immediate perceptual response to intersensory discrepancy.
Welch, R. B., & Warren, D. H. (1980) · Psychological Bulletin · doi:10.1037/0033-2909.88.3.638 · Cited by 1,176
Measurement and modeling of depth cue combination: in defense of weak fusion
Landy, M. S., Maloney, L. T., Johnston, E. B., & Young, M. (1995) · Vision Research · doi:10.1016/0042-6989(94)00176-m · Cited by 997
Integration of Proprioceptive and Visual Position-Information: An Experimentally Supported Model
van Beers, R. J., Sittig, A. C., & Gon, J. J. D. V. D. (1999) · Journal of Neurophysiology · doi:10.1152/jn.1999.81.3.1355 · Cited by 742
Vision and Touch: An Experimentally Created Conflict between the Two Senses
Rock, I., & Victor, J. (1964) · Science · doi:10.1126/science.143.3606.594 · Cited by 702
Visual Capture of Touch: Out-of-the-Body Experiences With Rubber Gloves.
Pavani, F., Spence, C., & Driver, J. (2000) · Psychological Science · doi:10.1111/1467-9280.00270 · Cited by 608
Adaptation, after-effect and contrast in the perception of curved lines.
Gibson, J. J. (1933) · Journal of Experimental Psychology · doi:10.1037/h0074626 · Cited by 599
Data Fusion for Sensory Information Processing Systems
Clark, J. J., & Yuille, A. L. (1990) · Springer eBooks · doi:10.1007/978-1-4757-2076-1 · Cited by 525
Visual capture produced by prism spectacles
Hay, J. C., Pick, H. L., & Ikeda, K. (1965) · Psychonomic Science · doi:10.3758/bf03343413 · Cited by 309
Touch can change visual slant perception
Ernst, M. O., Banks, M. S., & Bülthoff, H. H. (2000) · Nature Neuroscience · doi:10.1038/71140 · Cited by 236
Self-citationAuditory Spatial Receptive Fields Created by Multiplication
Peña, J. L., & Konishi, M. (2001) · Science · doi:10.1126/science.1059201 · Cited by 222
Heller, M. A., & Heller, M. A. (2013) · Psychology Press eBooks · doi:10.4324/9781315799629 · Cited by 215
Visual and tactual texture perception: Intersensory cooperation
Heller, M. A. (1982) · Perception & Psychophysics · doi:10.3758/bf03202657 · Cited by 200
Computational models of sensorimotor integration
Ghahramani, Z., Wolpert, D. M., & Jordan, M. I. (1997) · Advances in psychology · doi:10.1016/s0166-4115(97)80006-4 · Cited by 165
Lederman, S. J., & Abbott, S. G. (1981) · Journal of Experimental Psychology: Human Perception and Performance · doi:10.1037//0096-1523.7.4.902 · Cited by 160
Discrimination of planar surface slant from texture: human and ideal observers compared
Knill, D. C. (1998) · Vision Research · doi:10.1016/s0042-6989(97)00325-8 · Cited by 151
Efference and the conscious experience of perception.
Festinger, L., Ono, H., & Burnham, C. A. (1967) · Journal of Experimental Psychology · doi:10.1037/h0024766 · Cited by 122
Self-Organization, Computational Maps, and Motor Control
(1997) · Advances in psychology · doi:10.1016/s0166-4115(97)x8001-x · Cited by 116
Shape from texture: Ideal observers and human psychophysics
Blake, A., Bülthoff, H., & Sheinberg, D. L. (1993) · Vision Research · doi:10.1016/0042-6989(93)90037-w · Cited by 115
Synaesthesia in the normal limb
Mon-Williams, M. A., Wann, J. P., Jenkinson, M., & Rushton, K. (1997) · Proceedings of the Royal Society B Biological Sciences · doi:10.1098/rspb.1997.0139 · Cited by 89
Estimator Reliability and Distance Scaling in Stereoscopic Slant Perception
Backus, B. T., & Banks, M. S. (1999) · Perception · doi:10.1068/p2753 · Cited by 71
Self-citationExtra-retinal and perspective cues cause the small range of the induced effect
Banks, M. S., & Backus, B. T. (1998) · Vision Research · doi:10.1016/s0042-6989(97)00179-x · Cited by 69
Self-citationHaptic Dominance in Form Perception with Blurred Vision
Heller, M. A. (1983) · Perception · doi:10.1068/p120607 · Cited by 66
How Optimal Depth Cue Integration Depends on the Task
Schrater, P., & Kersten, D. (2000) · International Journal of Computer Vision · doi:10.1023/a:1026557704054 · Cited by 48
Visual capture of haptically judged depth
Singer, G., & Day, R. H. (1969) · Perception & Psychophysics · doi:10.3758/bf03209572 · Cited by 30
The Dominance of Touch by Vision: Sometimes Incomplete
Power, R. P. (1980) · Perception · doi:10.1068/p090457 · Cited by 26
Research works printed but not linked: 1
- Tastevin, J. En partant de l’experience d’Aristote. L’Encephale 1, 57–84 (1937).
Cited by
Whatever next? Predictive brains, situated agents, and the future of cognitive science
Clark, A. (2013) · Behavioral and Brain Sciences · doi:10.1017/s0140525x12000477 · Cited by 5,924
Faisal, A. A., Selen, L. P. J., & Wolpert, D. M. (2008) · Nature Reviews Neuroscience · doi:10.1038/nrn2258 · Cited by 2,883
The Bayesian brain: The role of uncertainty in neural coding and computation
Knill, D. C., & Pouget, A. (2004) · Trends in Neurosciences · doi:10.1016/j.tins.2004.10.007 · Cited by 2,588
Learning the value of information in an uncertain world
Behrens, T. E. J., Woolrich, M. W., Walton, M. E., & Rushworth, M. F. S. (2007) · Nature Neuroscience · doi:10.1038/nn1954 · Cited by 2,053
Bayesian integration in sensorimotor learning
Körding, K. P., & Wolpert, D. M. (2004) · Nature · doi:10.1038/nature02169 · Cited by 2,004
Metadata
- Authors
- Ernst, M. O., & Banks, M. S.
- Affiliations
- Max Planck Institute for Biological Cybernetics (DE)
- University of California, Berkeley (US)
- Venue
- Nature
- Published
- 2002-01
- Volume
- 415
- Issue
- 6870
- Pages
- 429-433
- ISSN
- 0028-0836 (ISSN-L)
- 1476-4687
- OpenAlex
- W1969090956
- Cited by
- 4,844
- References
- 25
- Citations by year
- 2026: 237 · 2025: 267 · 2024: 260 · 2023: 287 · 2022: 220 · 2021: 279 · 2020: 311 · 2019: 316 · 2018: 267 · 2017: 275 · 2016: 192 · 2015: 230 · 2014: 207 · 2013: 195 · 2012: 198
- FWCI
- 80.64
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- www.springer.com/tdm (tdm)
- Locations
- Nature (publishedVersion)
- PubMed (publishedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- Max Planck Institute for Plasma Physics (submittedVersion)
- PUB – Publications at Bielefeld University (Bielefeld University) (submittedVersion)
- Publikationen an der Universität Bielefeld (Universität Bielefeld) (submittedVersion)
- MPG.PuRe (Max Planck Society) (submittedVersion)
- Funders
- National Science Foundation
- National Institutes of Health
- Topics
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Multisensory perception and integration
- Life Sciences › Neuroscience › Cognitive Neuroscience › Tactile and Sensory Interactions
- Life Sciences › Neuroscience › Cognitive Neuroscience › Visual perception and processing mechanisms
- Keywords
- visual-haptic integration, cue combination, maximum likelihood estimation, statistical optimality, visual dominance, visual estimation, height perception, variance minimization, multisensory perception
- MeSH terms
- Adult, Humans, Models, Neurological, Sensory Thresholds, Size Perception, Touch
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.