Auditory Sensory Substitution is Intuitive and Automatic with Texture Stimuli
Open accessThe DOI matches its Crossref record; 28 of the 28 references it cites were found in a registry or an index.
Abstract
Millions of people are blind worldwide. Sensory substitution (SS) devices (e.g., vOICe) can assist the blind by encoding a video stream into a sound pattern, recruiting visual brain areas for auditory analysis via crossmodal interactions and plasticity. SS devices often require extensive training to attain limited functionality. In contrast to conventional attention-intensive SS training that starts with visual primitives (e.g., geometrical shapes), we argue that sensory substitution can be engaged efficiently by using stimuli (such as textures) associated with intrinsic crossmodal mappings. Crossmodal mappings link images with sounds and tactile patterns. We show that intuitive SS sounds can be matched to the correct images by naive sighted participants just as well as by intensively-trained participants. This result indicates that existing crossmodal interactions and amodal sensory cortical processing may be as important in the interpretation of patterns by SS as crossmodal plasticity (e.g., the strengthening of existing connections or the formation of new ones), especially at the earlier stages of SS usage. An SS training procedure based on crossmodal mappings could both considerably improve participant performance and shorten training times, thereby enabling SS devices to significantly expand blind capabilities.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures






Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“The participants' feedback or task performance is used as an indicator to assess the efficiency of the conversion method in the conventional assessment [17], [40].”
From Deep Learning-Based Optimization of Visual–Auditory Sensory Substitution (2023) · cites this paper as [40]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Journal of Consciousness Studies
(2022) · Journal of Consciousness Studies · doi:10.53765/20512201 · Cited by 6,832
Crossmodal correspondences: A tutorial review
Spence, C. (2011) · Attention, Perception, & Psychophysics · doi:10.3758/s13414-010-0073-7 · Cited by 1,618
Synaesthesia? A window into perception, thought and language
Ramachandran, V. S., & Hubbard, E. M. (2001) · Cited by 1,570
Vision substitution by tactile image projection
BACH-Y-RITA, P., COLLINS, C. C., SAUNDERS, F. A., WHITE, B., & SCADDEN, L. (1969) · Nature · doi:10.1038/221963a0 · Cited by 956
Preattentive processing in vision
Treisman, A. (1985) · Computer Vision, Graphics, and Image Processing · doi:10.1016/s0734-189x(85)80004-9 · Cited by 935
An experimental system for auditory image representations
Meijer, P. B. L. (1992) · IEEE Transactions on Biomedical Engineering · doi:10.1109/10.121642 · Cited by 900
Chapter 27 The metamodal organization of the brain
Pascual-Leone, A., & Hamilton, R. (2001) · Progress in Brain Research · doi:10.1016/s0079-6123(01)34028-1 · Cited by 540
Shape conveyed by visual-to-auditory sensory substitution activates the lateral occipital complex
Amedi, A., Stern, W. M., Camprodon, J. A., Bermpohl, F., Merabet, L., Rotman, S., Hemond, C., Meijer, P., & Pascual-Leone, A. (2007) · Nature Neuroscience · doi:10.1038/nn1912 · Cited by 451
Audiovisual Integration of Speech Falters under High Attention Demands
Alsius, A., Navarra, J., Campbell, R., & Soto-Faraco, S. (2005) · Current Biology · doi:10.1016/j.cub.2005.03.046 · Cited by 380
Form perception with a 49-point electrotactile stimulus array on the tongue: a technical note.
Bach‐y‐Rita, P., Kaczmarek, K. A., Tyler, M. E., & Garcia-Lara, J. (1998) · PubMed · Cited by 338
Ptito, M. (2005) · Brain · doi:10.1093/brain/awh380 · Cited by 310
Striem-Amit, E., Cohen, L., Dehaene, S., & Amedi, A. (2012) · Neuron · doi:10.1016/j.neuron.2012.08.026 · Cited by 294
Parise, C. V., & Spence, C. (2012) · Experimental Brain Research · doi:10.1007/s00221-012-3140-6 · Cited by 285
Auvray, M., Hanneton, S., & O'Regan, J. K. (2007) · Perception · doi:10.1068/p5631 · Cited by 268
Maidenbaum, S., Abboud, S., & Amedi, A. (2014) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2013.11.007 · Cited by 230
The spatial character of high and low tones.
Pratt, C. C. (1930) · Journal of Experimental Psychology · doi:10.1037/h0072651 · Cited by 216
Cross-modality matching of brightness and loudness.
Stevens, J. C., & Marks, L. E. (1965) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.54.2.407 · Cited by 182
Collignon, O., Lassonde, M., Lepore, F., Bastien, D., & Veraart, C. (2006) · Cerebral Cortex · doi:10.1093/cercor/bhj162 · Cited by 171
How automatic are crossmodal correspondences?
Spence, C., & Deroy, O. (2013) · Consciousness and Cognition · doi:10.1016/j.concog.2012.12.006 · Cited by 160
Navigation with a sensory substitution device in congenitally blind individuals
Chebat, D.-R., Schneider, F. C., Kupers, R., & Ptito, M. (2011) · Neuroreport · doi:10.1097/wnr.0b013e3283462def · Cited by 132
Visual-haptic cue weighting is independent of modality-specific attention
Helbig, H. B., & Ernst, M. O. (2008) · Journal of Vision · doi:10.1167/8.1.21 · Cited by 94
Proulx, M. J., Stoerig, P., Ludowig, E., & Knoll, I. (2008) · PLoS ONE · doi:10.1371/journal.pone.0001840 · Cited by 81
Functional recruitment of visual cortex for sound encoded object identification in the blind
Merabet, L. B., Battelli, L., Obretenova, S., Maguire, S., Meijer, P., & Pascual-Leone, A. (2009) · NeuroReport · doi:10.1097/wnr.0b013e32832104dc · Cited by 78
Interactive Coding of Visual Spatial Frequency and Auditory Amplitude-Modulation Rate
Guzman-Martinez, E., Ortega, L., Grabowecky, M., Mossbridge, J., & Suzuki, S. (2012) · Current Biology · doi:10.1016/j.cub.2012.01.004 · Cited by 78
Brown, D. J., Macpherson, T., & Ward, J. (2011) · Perception · doi:10.1068/p6952 · Cited by 77
What neuroimaging tells us about sensory substitution
Poirier, C., Volder, A. D., & Scheiber, C. (2007) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2007.05.010 · Cited by 70
Generalized Learning of Visual-to-Auditory Substitution in Sighted Individuals
Kim, J.-K., & Zatorre, R. J. (2008) · Brain Research · doi:10.1016/j.brainres.2008.06.038 · Cited by 62
Perceptual load influences auditory space perception in the ventriloquist aftereffect
Eramudugolla, R., Kamke, M. R., Soto‐Faraco, S., & Mattingley, J. B. (2010) · Cognition · doi:10.1016/j.cognition.2010.09.009 · Cited by 32
Research works printed but not linked: 1
- Reynolds, Z. & Glenney, B. in Proceedings of Asia-Pacific Computing and Philosophy. 131–134 (2009).
Cited by
Designing sensory-substitution devices: Principles, pitfalls and potential1
Kristjánsson, Á., Moldoveanu, A., Jóhannesson, Ó. I., Balan, O., Spagnol, S., Valgeirsdóttir, V. V., & Unnthorsson, R. (2016) · Restorative Neurology and Neuroscience · doi:10.3233/rnn-160647 · Cited by 116
“Note that sensory substitution may not require training if crossmodal mappings are strategically utilized (Stiles & Shimojo, 2015).”
BackgroundIs Multimedia Multisensorial? - A Review of Mulsemedia Systems
Covaci, A., Zou, L., Tal, I., Muntean, G., & Ghinea, G. (2018) · ACM Computing Surveys · doi:10.1145/3233774 · Cited by 126
“As a result, the process becomes more intuitive, training time is shortened, and the participant performance is improved [192].”
BackgroundEXTENDED REALITY (XR) ENVIRONMENTS
Stanney, K. M., Nye, H., Haddad, S. D., Hale, K. S., Padron, C. K., & Cohn, J. V. (2021) · Wiley eBooks · doi:10.1002/9781119636113.ch30 · Cited by 93
Henschke, J. U., Oelschlegel, A. M., Angenstein, F., Ohl, F. W., Goldschmidt, J., Kanold, P. O., & Budinger, E. (2017) · Brain Structure and Function · doi:10.1007/s00429-017-1549-1 · Cited by 80
“…of the critical period, for example, bymultisensoryperceptual training (e.g., Proulx et al. 2014; McGovern et al. 2016), sensory substitution (e.g., Stiles and Shimojo 2015; Maidenbaum et al. 2014), or, though still prospectively, manipulation of the molecular background (e.g., Berardi et al.…”
Multisensory inclusive design with sensory substitution
Esenkaya, T., Lloyd‐Esenkaya, V., O'Neill, E. J., & Proulx, M. J. (2020) · Cognitive Research Principles and Implications · doi:10.1186/s41235-020-00240-7 · Cited by 94
Metadata
- Authors
- Stiles, N. R. B., & Shimojo, S.
- Affiliations
- California Institute of Technology (US)
- Corresponding author
- Stiles, Noelle R. B.
- ORCID
- Stiles, Noelle R. B.
- Venue
- Scientific Reports
- Published
- 2015-10-22
- Volume
- 5
- Issue
- 1
- Pages
- 15628-15628
- ISSN
- 2045-2322 (ISSN-L)
- PMC
- PMC4615028
- OpenAlex
- W1822494353
- Cited by
- 45
- References
- 28
- Citations by year
- 2026: 1 · 2025: 3 · 2024: 2 · 2023: 3 · 2022: 5 · 2021: 7 · 2020: 7 · 2019: 2 · 2018: 8 · 2017: 4 · 2016: 3
- FWCI
- 3.01
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- creativecommons.org/licenses/by/4.0 (tdm)
- creativecommons.org/licenses/by/4.0 (vor)
- Locations
- Scientific Reports (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- CaltechAUTHORS (California Institute of Technology) (submittedVersion, cc-by)
- Funders
- National Science Foundation
- Japan Science and Technology Agency
- Core Research for Evolutional Science and Technology
- Topics
- Life Sciences › Neuroscience › Cognitive Neuroscience › Tactile and Sensory Interactions
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Multisensory perception and integration
- Health Sciences › Health Professions › Occupational Therapy › Assistive Technology in Communication and Mobility
- Keywords
- sensory substitution, texture perception, auditory perception, visual impairment, blind users, voice, cross-modal plasticity, training efficiency
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.