Auditory Sensory Substitution is Intuitive and Automatic with Texture Stimuli
Open accessThe DOI matches its Crossref record; 28 of the 28 references it cites were found in a registry or an index.
Abstract
Millions of people are blind worldwide. Sensory substitution (SS) devices (e.g., vOICe) can assist the blind by encoding a video stream into a sound pattern, recruiting visual brain areas for auditory analysis via crossmodal interactions and plasticity. SS devices often require extensive training to attain limited functionality. In contrast to conventional attention-intensive SS training that starts with visual primitives (e.g., geometrical shapes), we argue that sensory substitution can be engaged efficiently by using stimuli (such as textures) associated with intrinsic crossmodal mappings. Crossmodal mappings link images with sounds and tactile patterns. We show that intuitive SS sounds can be matched to the correct images by naive sighted participants just as well as by intensively-trained participants. This result indicates that existing crossmodal interactions and amodal sensory cortical processing may be as important in the interpretation of patterns by SS as crossmodal plasticity (e.g., the strengthening of existing connections or the formation of new ones), especially at the earlier stages of SS usage. An SS training procedure based on crossmodal mappings could both considerably improve participant performance and shorten training times, thereby enabling SS devices to significantly expand blind capabilities.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures






Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“The participants' feedback or task performance is used as an indicator to assess the efficiency of the conversion method in the conventional assessment [17], [40].”
From Deep Learning-Based Optimization of Visual–Auditory Sensory Substitution (2023) · cites this paper as [40]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Journal of Consciousness Studies
(2022) · Journal of Consciousness Studies · doi:10.53765/20512201 · Cited by 6,832
Crossmodal correspondences: A tutorial review
Spence, C. (2011) · Attention, Perception, & Psychophysics · doi:10.3758/s13414-010-0073-7 · Cited by 1,618
Synaesthesia? A window into perception, thought and language
Ramachandran, V. S., & Hubbard, E. M. (2001) · Cited by 1,570
Vision substitution by tactile image projection
BACH-Y-RITA, P., COLLINS, C. C., SAUNDERS, F. A., WHITE, B., & SCADDEN, L. (1969) · Nature · doi:10.1038/221963a0 · Cited by 956
Preattentive processing in vision
Treisman, A. (1985) · Computer Vision, Graphics, and Image Processing · doi:10.1016/s0734-189x(85)80004-9 · Cited by 935
An experimental system for auditory image representations
Meijer, P. B. L. (1992) · IEEE Transactions on Biomedical Engineering · doi:10.1109/10.121642 · Cited by 900
Chapter 27 The metamodal organization of the brain
Pascual-Leone, A., & Hamilton, R. (2001) · Progress in Brain Research · doi:10.1016/s0079-6123(01)34028-1 · Cited by 540
Shape conveyed by visual-to-auditory sensory substitution activates the lateral occipital complex
Amedi, A., Stern, W. M., Camprodon, J. A., Bermpohl, F., Merabet, L., Rotman, S., Hemond, C., Meijer, P., & Pascual-Leone, A. (2007) · Nature Neuroscience · doi:10.1038/nn1912 · Cited by 451
Audiovisual Integration of Speech Falters under High Attention Demands
Alsius, A., Navarra, J., Campbell, R., & Soto-Faraco, S. (2005) · Current Biology · doi:10.1016/j.cub.2005.03.046 · Cited by 380
Form perception with a 49-point electrotactile stimulus array on the tongue: a technical note.
Bach‐y‐Rita, P., Kaczmarek, K. A., Tyler, M. E., & Garcia-Lara, J. (1998) · PubMed · Cited by 338
Research works printed but not linked: 1
- Reynolds, Z. & Glenney, B. in Proceedings of Asia-Pacific Computing and Philosophy. 131–134 (2009).
Cited by
Designing sensory-substitution devices: Principles, pitfalls and potential1
Kristjánsson, Á., Moldoveanu, A., Jóhannesson, Ó. I., Balan, O., Spagnol, S., Valgeirsdóttir, V. V., & Unnthorsson, R. (2016) · Restorative Neurology and Neuroscience · doi:10.3233/rnn-160647 · Cited by 116
“Note that sensory substitution may not require training if crossmodal mappings are strategically utilized (Stiles & Shimojo, 2015).”
BackgroundIs Multimedia Multisensorial? - A Review of Mulsemedia Systems
Covaci, A., Zou, L., Tal, I., Muntean, G., & Ghinea, G. (2018) · ACM Computing Surveys · doi:10.1145/3233774 · Cited by 126
“As a result, the process becomes more intuitive, training time is shortened, and the participant performance is improved [192].”
BackgroundEXTENDED REALITY (XR) ENVIRONMENTS
Stanney, K. M., Nye, H., Haddad, S. D., Hale, K. S., Padron, C. K., & Cohn, J. V. (2021) · Wiley eBooks · doi:10.1002/9781119636113.ch30 · Cited by 93
Henschke, J. U., Oelschlegel, A. M., Angenstein, F., Ohl, F. W., Goldschmidt, J., Kanold, P. O., & Budinger, E. (2017) · Brain Structure and Function · doi:10.1007/s00429-017-1549-1 · Cited by 80
“…of the critical period, for example, bymultisensoryperceptual training (e.g., Proulx et al. 2014; McGovern et al. 2016), sensory substitution (e.g., Stiles and Shimojo 2015; Maidenbaum et al. 2014), or, though still prospectively, manipulation of the molecular background (e.g., Berardi et al.…”
Multisensory inclusive design with sensory substitution
Esenkaya, T., Lloyd‐Esenkaya, V., O'Neill, E. J., & Proulx, M. J. (2020) · Cognitive Research Principles and Implications · doi:10.1186/s41235-020-00240-7 · Cited by 94
SoundSight: a mobile sensory substitution device that sonifies colour, distance, and temperature
Hamilton-Fletcher, G., Alvarez, J., Obrist, M., & Ward, J. (2022) · Journal on Multimodal User Interfaces · doi:10.1007/s12193-021-00376-w · Cited by 28
“In this approach, basic shapes and textures remain intuitive for users [14, 16], but users can have difficulty in counting and tracking multiple simultaneous frequencies in a soundscape [14, 30, 31].”
BackgroundSensory augmentation: integration of an auditory compass signal into human perception of space
Schumann, F., & O’Regan, J. K. (2017) · Scientific Reports · doi:10.1038/srep42197 · Cited by 25
Expanding the sense of touch outside the body
Berger, C. C., & González-Franco, M. (2018) · ACM Symposium on Applied Perception · doi:10.1145/3225153.3225172 · Cited by 24
Hu, D., Wang, D., Li, X., Nie, F., & Wang, Q. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00816 · Cited by 21
“The blind participants’ feedback or task performance is usually served as the indicator to the quality of encoding schemes in the conventional assessment [3, 33].”
BackgroundMethodInfluentialNew frontiers of retinal therapeutic intervention: a critical analysis of novel approaches
Nanegrungsunk, O., Au, A., Sarraf, D., & Sadda, S. R. (2022) · Annals of Medicine · doi:10.1080/07853890.2022.2066169 · Cited by 20
Multi-Sensory Color Code Based on Sound and Scent for Visual Art Appreciation
Quero, L. C., Lee, C.-H., & Cho, J. (2021) · Electronics · doi:10.3390/electronics10141696 · Cited by 24
Stiles, N. R. B., Patel, V. R., & Weiland, J. D. (2021) · Vision Research · doi:10.1016/j.visres.2021.01.008 · Cited by 20
Self-citationChow, H. M., & Ciaramitaro, V. M. (2018) · Journal of Experimental Child Psychology · doi:10.1016/j.jecp.2018.10.005 · Cited by 17
Testing vision is not testing for vision
Peli, E. (2020) · Translational Vision Science & Technology · doi:10.1167/tvst.9.13.32 · Cited by 14
Multi-modal cognitive computing
李, 学. (2022) · Scientia Sinica Informationis · doi:10.1360/ssi-2022-0226 · Cited by 14
Bordeau, C., Scalvini, F., Migniot, C., Dubois, J., & Ambard, M. (2023) · Frontiers in Psychology · doi:10.3389/fpsyg.2023.1079998 · Cited by 14
“In the SSD research, it has been suggested that the pitch-based elevation mapping is intuitive in an object recognition task (Stiles and Shimojo, 2015).”
BackgroundMethodDeep Learning-Based Optimization of Visual–Auditory Sensory Substitution
Kim, M., Park, Y., Moon, K., & Jeong, C. Y. (2023) · IEEE Access · doi:10.1109/access.2023.3243641 · Cited by 12
Usability assessment of assistive technology for blind and visually impaired
Dascălu, M.-I., Moldoveanu, A. D. B., Balan, O. C., Lupu, R. G., Ungureanu, F., & Caraiman, S. (2017) · E-Health and Bioengineering Conference (EHB) · doi:10.1109/ehb.2017.7995476 · Cited by 12
A self-training program for sensory substitution devices
Buchs, G., Haimler, B., Kerem, M., Maidenbaum, S., Braun, L., & Amedi, A. (2021) · PLoS ONE · doi:10.1371/journal.pone.0250281 · Cited by 14
“Note that this basic training is still required despite SSD main features being based on cross-modal correspondences, which potentially allow a certain degree of intuitive learning in the users making this stage faster and easier [3, 15, 16].”
BackgroundResultImproved Spatial Knowledge Acquisition through Sensory Augmentation
Schmidt, V., König, S. U., Dilawar, R., Pacheco, T. S., & König, P. (2023) · Brain Sciences · doi:10.3390/brainsci13050720 · Cited by 11
Perception and Interactive Technology
Scheller, M., Petrini, K., & Proulx, M. J. (2018) · Wiley eBooks · doi:10.1002/9781119170174.epcn215 · Cited by 12
Łukowska, M., Kałwak, W., Osiński, D., Janik, J., & Wierzchoń, M. (2022) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2022.102925 · Cited by 11
Mixing up the Senses: Sensory Substitution Is Not a Form of Artificially Induced Synaesthesia
Kirsch, L. P., Job, X., & Auvray, M. (2020) · Multisensory Research · doi:10.1163/22134808-bja10010 · Cited by 8
“Some have suggested that after training, SSD use becomes automatic (Stiles and Shimojo, 2015) and that from this moment the device and the sensory processing becomes ‘transparent’.”
BackgroundMaimon, A., Wald, I. Y., Snir, A., Oz, M. B., & Amedi, A. (2024) · PLoS ONE · doi:10.1371/journal.pone.0310033 · Cited by 10
Sensory augmentation to aid training with retinal prostheses
Kvansakul, J. G. S., Hamilton, L. H. W., Ayton, L. N., McCarthy, C. D., & Petoe, M. A. (2020) · Journal of Neural Engineering · doi:10.1088/1741-2552/ab9e1d · Cited by 7
“Natural textures were found to be easy to interpret by naïve users despite their complexity, implying they could be used alongside simpler images during initial training, providing a more authentic experience [58].”
BackgroundHamilton-Fletcher, G., & Chan, K. C. (2021) · Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC) · doi:10.1109/embc46164.2021.9630296 · Cited by 11
“Stiles and Shimojo [15] showed that while subjects were good at”
BackgroundResultAutoencoding sensory substitution
Tóth, V., & Parkkonen, L. (2019) · Unpublished · doi:10.13140/rg.2.2.10576.87048 · Cited by 7
“Furthermore, implicit training suffices for SS of simple, abstract shapes [32, 162].”
BackgroundJiang, P., Kent, C. D., & Rossiter, J. M. (2024) · PLoS ONE · doi:10.1371/journal.pone.0299213 · Cited by 10
Enhancing the Fashion and Textile Design Process and Wearer Experiences
Moody, W., Langdon, P., & Karam, M. (2018) · Springer eBooks · doi:10.1007/978-3-319-75028-6_5 · Cited by 6
Vincent, M., Tang, H., Khoo, W. L., Zhu, Z., & Ro, T. (2016) · Multisensory Research · doi:10.1163/22134808-00002542 · Cited by 7
“On the other hand, several recent studies have demonstrated that the vOICe (Brown et al., 2015; Stiles and Shimojo, 2015) and EyeMusic (Maidenbaum et al., 2016), two different auditory SSDs, are intuitive to use, with high levels of performance after minimal or short training.”
BackgroundCan perception be extended to a “feel of north”? Tests of automaticity with the NaviEar
Witzel, C., Lübbert, A., O'Regan, J. K., Hanneton, S., & Schumann, F. (2022) · Adaptive Behavior · doi:10.1177/10597123221130235 · Cited by 8
“…have measured some aspects of automaticity in the use of sensory augmentation and sensory substitution devices (Elliott et al., 2010; Kaspar et al., 2014; K¨onig et al., 2016; Nagel et al., 2005; Pielot & Boll, 2010; Schumann & O ’ Regan, 2017; Stiles & Shimojo, 2015; Van Erp & Werkhoven, 2006).”
InfluentialRuan, L., Hamilton-Fletcher, G., Beheshti, M., Hudson, T. E., Porfiri, M., & Rizzo, J. (2025) · Disability and Rehabilitation Assistive Technology · doi:10.1080/17483107.2025.2463541 · Cited by 8
“…of practice by pBlV to be mastered for functional tasks [44]. these forms of sensory substitution may be intuitive for simple shapes or patterns [45] but typically require months of training to accurately interpret raw visual images of more complicated imagery [19]. however, simplifying complex…”
Word recognition: re-thinking prosthetic vision evaluation
Han, S., Qiu, C., Lee, K. R., Jung, J., & Peli, E. (2018) · Journal of Neural Engineering · doi:10.1088/1741-2552/aac663 · Cited by 4
“…provide non visual input through tactile sensations on the skin (Meers and Ward 2005, Ortiz et al 2011) or tongue (Grant et al 2016), or through auditory stimulations (Capelle et al 1998, Cronly-Dillon et al 1999, Hanneton et al 2010, Striem-Amit et al 2012a, 2012b, Stiles and Shimojo 2015).”
BackgroundOrientation of tactile attention on the surface of the tongue.
Richardson, M., Petrini, K., & Proulx, M. J. (2022) · Journal of Experimental Psychology Human Perception & Performance · doi:10.1037/xhp0001007 · Cited by 6
Bordeau, C., Scalvini, F., Migniot, C., Dubois, J., & Ambard, M. (2025) · Attention Perception & Psychophysics · doi:10.3758/s13414-025-03065-y · Cited by 3
“…that blindfolded sighted participants find the use of pitch modulation for the elevation dimension in SSD to be intuitive (Bordeau et al., 2023; Stiles & Shimojo, 2015), it has been suggested that the cross-modal correspondence between pitch height and spatial height is weaker in the blind…”
Software Design for Low-Latency Visuo-Auditory Sensory Substitution on Mobile Devices
Ambard, M. (2017) · Computer and Information Science · doi:10.5539/cis.v10n2p1 · Cited by 4
“Recent work has shown that sounds can be efficiently used as a substitute for visual and tactile textures (Stiles & Shimojo, 2015).”
BackgroundMethodZempo, K., Mashiba, Y., Kawamura, T., Kuratomo, N., & Salih, H. E. B. (2018) · ACM Symposium on User Interface Software and Technology (UIST) · doi:10.1145/3266037.3266120 · Cited by 3
“…visual impaired to understand and distinguish photographs similar to this research purpose, the method of making images audible as spectrograms [10] and the method of recognizing and identifying photographs and reading out what kind of scene [11] useful for recognition and identification of…”
MethodCan perception be extended to a “feel of North”? Tests of automaticity with the NaviEarPreprint
Witzel, C., Lübbert, A., Schumann, F., Hanneton, S., & O'Regan, J. K. (2021) · doi:10.31234/osf.io/2yk8b · Cited by 2
Jiang, P., Rossiter, J. M., & Kent, C. D. (2025) · PLoS ONE · doi:10.1371/journal.pone.0318354 · Cited by 2
“Implementing stimuli connected to intrinsic cross-modal mapping can contribute to achieving effective sensory substitution [25].”
Earballs: Neural Transmodal Translation.Preprint
Port, A., Kim, C., & Patel, M. (2020) · arXiv (Cornell University) · Cited by 1
Baez, G., Prieto, P. A., & Cheein, F. A. (2018) · Biomedical Physics & Engineering Express · doi:10.1088/2057-1976/aac9ad
Design for Medical Simulation: Guidelines and Visioning for a New Model of Education
Palmieri, S., Bisson, M., Zinzone, M., & Ianniello, A. (2020) · Lecture notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering · doi:10.1007/978-3-030-60036-5_20
Preparing participants for the use of the tongue visual sensory substitution device
Pamir, Z., Jung, J., & Peli, E. (2020) · Disability and Rehabilitation Assistive Technology · doi:10.1080/17483107.2020.1821102
Audio-Tactile School-Based Yoga for Visually Impaired Children
Das, P., Saha, S., Kanojiya, R., Chetry, P., & Choudhary, P. K. (2026) · IGI Global eBooks · doi:10.4018/979-8-2600-1066-2.ch011
Metadata
- Authors
- Stiles, N. R. B., & Shimojo, S.
- Affiliations
- California Institute of Technology (US)
- Corresponding author
- Stiles, Noelle R. B.
- ORCID
- Stiles, Noelle R. B.
- Venue
- Scientific Reports
- Published
- 2015-10-22
- Volume
- 5
- Issue
- 1
- Pages
- 15628-15628
- ISSN
- 2045-2322 (ISSN-L)
- PMC
- PMC4615028
- OpenAlex
- W1822494353
- Cited by
- 45
- References
- 28
- Citations by year
- 2026: 1 · 2025: 3 · 2024: 2 · 2023: 3 · 2022: 5 · 2021: 7 · 2020: 7 · 2019: 2 · 2018: 8 · 2017: 4 · 2016: 3
- FWCI
- 3.01
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- creativecommons.org/licenses/by/4.0 (tdm)
- creativecommons.org/licenses/by/4.0 (vor)
- Locations
- Scientific Reports (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- CaltechAUTHORS (California Institute of Technology) (submittedVersion, cc-by)
- Funders
- National Science Foundation
- Japan Science and Technology Agency
- Core Research for Evolutional Science and Technology
- Topics
- Life Sciences › Neuroscience › Cognitive Neuroscience › Tactile and Sensory Interactions
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Multisensory perception and integration
- Health Sciences › Health Professions › Occupational Therapy › Assistive Technology in Communication and Mobility
- Keywords
- sensory substitution, texture perception, auditory perception, visual impairment, blind users, voice, cross-modal plasticity, training efficiency
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.