Learning to integrate arbitrary signals from vision and touch
Open accessAbstract
When different perceptual signals of the same physical property are integrated, for example, an objects' size, which can be seen and felt, they form a more reliable sensory estimate (e.g., M. O. Ernst & M. S. Banks, 2002). This, however, implies that the sensory system already knows which signals belong together and how they relate. In other words, the system has to know the mapping between the signals. In a Bayesian model of cue integration, this prior knowledge can be made explicit. Here, we ask whether such a mapping between two arbitrary sensory signals from vision and touch can be learned from their statistical co-occurrence such that they become integrated. In the Bayesian framework, this means changing the belief about the distribution of the stimuli. To this end, we trained subjects with stimuli that are usually unrelated in the world--the luminance of an object (visual signal) and its stiffness (haptic signal). In the training phase, we then presented subjects with combinations of these two signals, which were artificially correlated, and thus, we introduced a new mapping between them. For example, the stiffer the object, the brighter it was. We measured the influence of learning by comparing discrimination performance before and after training. The prediction is that integration makes discrimination worse for stimuli, which are incongruent with the newly learned mapping, because integration would cause this incongruency to disappear perceptually. The more certain subjects are about the new mapping, the stronger should the influence be on discrimination performance. Thus, learning in this context is about acquiring beliefs. We found a significant change in discrimination performance before and after training when comparing trials with congruent and incongruent stimuli. After training, discrimination thresholds for the incongruent stimuli are increased relative to thresholds for congruent stimuli, suggesting that subjects learned effectively to integrate the two formerly unrelated signals.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
According to Bayesian decision theory, the brain uses prior knowledge about what sensory signals go together to integrate appropriate sensory stimuli to maximise reliability of perceived information and reduce cognitive load.
“According to Bayesian decision theory, the brain uses prior knowledge about what sensory signals go together-whether inborn or explicitly learned-to integrate appropriate sensory stimuli with the goal of maximising the reliability of perceived information [139-141] and, presumably, to reduce cognitive load by combining disparate sensory cues into a single object.”
From The Role of Intrinsic and Extrinsic Sensory Factors in Sweetness Perception of Food and Beverages: A Review (2019) · cites this paper as [139]
Cites
Humans integrate visual and haptic information in a statistically optimal fashion
Ernst, M. O., & Banks, M. S. (2002) · Nature · doi:10.1038/415429a · Cited by 4,844 · see citing works
Self-citationThe psychometric function: I. Fitting, sampling, and goodness of fit
Wichmann, F. A., & Hill, N. J. (2001) · Perception & Psychophysics · doi:10.3758/bf03194544 · Cited by 2,359 · see citing works
Handbuch der physiologischen Optik
Helmholtz, H. L. F. (2011) · DFG Viewer (DFG Viewer) · Cited by 1,947 · see citing works
The Ventriloquist Effect Results from Near-Optimal Bimodal Integration
Alais, D., & Burr, D. (2004) · Current Biology · doi:10.1016/j.cub.2004.01.029 · Cited by 1,936 · see citing works
Merging the senses into a robust percept
Ernst, M. O., & Bülthoff, H. H. (2004) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2004.02.002 · Cited by 1,906 · see citing works
Self-citationPerception as Bayesian Inference
(1996) · doi:10.1017/cbo9780511984037 · Cited by 1,425 · see citing works
Measurement and modeling of depth cue combination: in defense of weak fusion
Landy, M. S., Maloney, L. T., Johnston, E. B., & Young, M. (1995) · Vision Research · doi:10.1016/0042-6989(94)00176-m · Cited by 996 · see citing works
Goodwin, C. (1995) · Social Studies of Science · doi:10.1177/030631295025002002 · Cited by 620 · see citing works
Modality-Constrained Statistical Learning of Tactile, Visual, and Auditory Sequences.
Conway, C. M., & Christiansen, M. H. (2005) · Journal of Experimental Psychology Learning Memory and Cognition · doi:10.1037/0278-7393.31.1.24 · Cited by 538 · see citing works
Data Fusion for Sensory Information Processing Systems
Clark, J. J., & Yuille, A. (1990) · Springer eBooks · doi:10.1007/978-1-4757-2076-1 · Cited by 519 · see citing works
Combining Sensory Information: Mandatory Fusion Within, but Not Between, Senses
Hillis, J. M., Ernst, M. O., Banks, M. S., & Landy, M. S. (2002) · Science · doi:10.1126/science.1075396 · Cited by 486 · see citing works
Self-citationSlant from texture and disparity cues: Optimal cue combination
Hillis, J. M., Watt, S. J., Landy, M. S., & Banks, M. S. (2004) · Journal of Vision · doi:10.1167/4.12.1 · Cited by 421 · see citing works
Do humans optimally integrate stereo and texture information for judgments of surface slant?
Knill, D. C., & Saunders, J. A. (2003) · Vision Research · doi:10.1016/s0042-6989(03)00458-9 · Cited by 403 · see citing works
The statistical determinants of adaptation rate in human reaching
Burge, J., Ernst, M. O., & Banks, M. S. (2008) · Journal of Vision · doi:10.1167/8.4.20 · Cited by 400 · see citing works
Self-citationExperience can change the 'light-from-above' prior
Adams, W. J., Graf, E. W., & Ernst, M. O. (2004) · Nature Neuroscience · doi:10.1038/nn1312 · Cited by 377 · see citing works
Self-citationHuman Body Perception From The Inside Out
(2005) · doi:10.1093/oso/9780195178371.001.0001 · Cited by 339 · see citing works
Integration of depth modules: stereo and shading
Bülthoff, H., & Mallot, H. A. (1988) · Journal of the Optical Society of America A · doi:10.1364/josaa.5.001749 · Cited by 333 · see citing works
Statistical Learning Within and Between Modalities
Conway, C. M., & Christiansen, M. H. (2006) · Psychological Science · doi:10.1111/j.1467-9280.2006.01801.x · Cited by 295 · see citing works
Sound-induced flash illusion as an optimal percept
Shams, L. B., Ma, W. J., & Beierholm, U. (2005) · Neuroreport · doi:10.1097/01.wnr.0000187634.68504.bb · Cited by 267 · see citing works
Optimal integration of shape information from vision and touch
Helbig, H. B., & Ernst, M. O. (2007) · Experimental Brain Research · doi:10.1007/s00221-006-0814-y · Cited by 266 · see citing works
Self-citationBayesian decision theory and psychophysics
Yuille, A. L., & Bülthoff, H. H. (1996) · Perception as Bayesian Inference · doi:10.1017/cbo9780511984037.006 · Cited by 250 · see citing works
Roach, N. W., Heron, J., & McGraw, P. V. (2006) · Proceedings of the Royal Society B Biological Sciences · doi:10.1098/rspb.2006.3578 · Cited by 210 · see citing works
A Bayesian View on Multimodal Cue Integration
Ernst, M. O. (2005) · Human Body Perception From The Inside Out · doi:10.1093/oso/9780195178371.003.0006 · Cited by 209 · see citing works
Self-citationKnill, D. C. (2007) · Journal of Vision · doi:10.1167/7.7.5 · Cited by 191 · see citing works
What determines visual cue reliability?
Jacobs, R. A. (2002) · Trends in Cognitive Sciences · doi:10.1016/s1364-6613(02)01948-4 · Cited by 182 · see citing works
Vision and touch are automatically integrated for the perception of sequences of events
Bresciani, J., Dammeier, F., & Ernst, M. O. (2006) · Journal of Vision · doi:10.1167/6.5.2 · Cited by 180 · see citing works
Self-citationComputational models of sensorimotor integration
Zoubin Ghahramani, A., Wolpert, D. M., & Jordan, M. I. (1997) · Advances in psychology · doi:10.1016/s0166-4115(97)80006-4 · Cited by 159 · see citing works
Ideal cue combination for localizing texture-defined edges
Landy, M. S., & Kojima, H. (2001) · Journal of the Optical Society of America A · doi:10.1364/josaa.18.002307 · Cited by 137 · see citing works
Self-Organization, Computational Maps, and Motor Control
(1997) · Advances in psychology · doi:10.1016/s0166-4115(97)x8001-x · Cited by 116 · see citing works
Demonstration of cue recruitment: Change in visual appearance by means of Pavlovian conditioning
Qi, H., Saunders, J. A., Stone, R. W., & Backus, B. T. (2005) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.0506728103 · Cited by 112 · see citing works
Mixture models and the probabilistic structure of depth cues.
Knill, D. (2003) · Vision Research · doi:10.1016/s0042-6989(03)00003-8 · Cited by 108 · see citing works
Adaptation to three-dimensional distortions in human vision
Adams, W. J., Banks, M. S., & van Ee, R. (2001) · Nature Neuroscience · doi:10.1038/nn729 · Cited by 84 · see citing works
Probabilistic Models of the Brain.
Long, D. M. (2002) · Neurosurgery Quarterly · doi:10.1097/00013414-200212000-00005 · Cited by 67 · see citing works
The optimum decision rules for the oddity task
Versfeld, N. J., Dai, H., & Green, D. M. (1996) · Perception & Psychophysics · doi:10.3758/bf03205470 · Cited by 30 · see citing works
Learning to Combine Arbitrary Signals from Vision and Touch
Jäkel, F., & Ernst, M. O. (2003) · TUbilio (Technical University of Darmstadt) · Cited by 8 · see citing works
Self-citationInteractions between depth cues
Howard, I. P., & Rogers, B. (2008) · Oxford University Press eBooks · doi:10.1093/acprof:oso/9780195367607.003.0013 · Cited by 4 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Crossmodal correspondences: A tutorial review
Spence, C. (2011) · Attention, Perception, & Psychophysics · doi:10.3758/s13414-010-0073-7 · Cited by 1,618 · see citing works
Benefits of multisensory learning
Shams, L., & Seitz, A. R. (2008) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2008.07.006 · Cited by 1,178 · see citing works
A Bayesian perspective on magnitude estimation
Petzschner, F. H., Glasauer, S., & Stephan, K. E. (2015) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2015.03.002 · Cited by 327 · see citing works
Parise, C. V., & Spence, C. (2009) · PLoS ONE · doi:10.1371/journal.pone.0005664 · Cited by 272 · see citing works
Wang, Q. J., Mielby, L. A., Junge, J. Y., Bertelsen, A. S., Kidmose, U., Spence, C., & Byrne, D. V. (2019) · Foods · doi:10.3390/foods8060211 · Cited by 168 · see citing works
Oral referral: On the mislocalization of odours to the mouth
Spence, C. (2016) · Food Quality and Preference · doi:10.1016/j.foodqual.2016.02.006 · Cited by 80 · see citing works
How multisensory neurons solve causal inference.
Rideaux, R., Storrs, K. R., Maiello, G., & Welchman, A. E. (2021) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2106235118 · Cited by 65 · see citing works
Peri-personal Space as a Prior in Coupling Visual and Proprioceptive Signals
Noel, J.-P., Samad, M., Doxon, A., Clark, J., Keller, S., & Di Luca, M. (2018) · Scientific Reports · doi:10.1038/s41598-018-33961-3 · Cited by 39 · see citing works
- Authors
- Ernst, M. O.
- Affiliations
- Max Planck Institute for Biological Cybernetics (DE)
- Corresponding author
- Ernst, Marc O.
- ORCID
- Ernst, Marc O.
- Venue
- Journal of Vision
- Published
- 2007-06-25
- Volume
- 7
- Issue
- 5
- Pages
- 7
- ISSN
- 1534-7362 (ISSN-L)
- OpenAlex
- W2148504296
- Cited by
- 332
- References
- 36
- Citations by year
- 2026: 9 · 2025: 11 · 2024: 15 · 2023: 11 · 2022: 20 · 2021: 20 · 2020: 23 · 2019: 21 · 2018: 16 · 2017: 18 · 2016: 17 · 2015: 20 · 2014: 15 · 2013: 28 · 2012: 21
- FWCI
- 21.32
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Locations
- Journal of Vision (publishedVersion)
- PubMed (publishedVersion)
- Max Planck Institute for Plasma Physics (submittedVersion)
- PUB – Publications at Bielefeld University (Bielefeld University) (submittedVersion)
- Publikationen an der Universität Bielefeld (Universität Bielefeld) (submittedVersion)
- MPG.PuRe (Max Planck Society) (submittedVersion)
- Funders
- European Commission
- Topics
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Multisensory perception and integration
- Life Sciences › Neuroscience › Cognitive Neuroscience › Tactile and Sensory Interactions
- Life Sciences › Neuroscience › Cognitive Neuroscience › Visual perception and processing mechanisms
- Keywords
- multisensory integration, prior learning, signal mapping, congruency effect, discrimination thresholds, perceptual learning, statistical learning, haptic perception
- MeSH terms
- Adult, Bayes Theorem, Cues, Differential Threshold, Discrimination, Psychological, Female, Form Perception, Humans, Learning, Male, Models, Psychological, Touch, Vision, Ocular
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
