EyeMusic: Introducing a “visual” colorful experience for the blind using auditory sensory substitution
Open accessThe DOI matches its Crossref record; 42 of the 43 references it cites were found in a registry or an index.
Abstract
Purpose: Sensory-substitution devices (SSDs) provide auditory or tactile representations of visual information. These devices often generate unpleasant sensations and mostly lack color information. We present here a novel SSD aimed at addressing these issues. Methods: We developed the EyeMusic, a novel visual-to-auditory SSD for the blind, providing both shape and color information. Our design uses musical notes on a pentatonic scale generated by natural instruments to convey the visual information in a pleasant manner. A short behavioral protocol was utilized to train the blind to extract shape and color information, and test their acquired abilities. Finally, we conducted a survey and a comparison task to assess the pleasantness of the generated auditory stimuli. Results: We show that basic shape and color information can be decoded from the generated auditory stimuli. High performance levels were achieved by all participants following as little as 2–3 hours of training. Furthermore, we show that users indeed found the stimuli pleasant and potentially tolerable for prolonged use. Conclusions: The novel EyeMusic algorithm provides an intuitive and relatively pleasant way for the blind to extract shape and color information. We suggest that this might help facilitating visual rehabilitation because of the added functionality and enhanced pleasantness.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Using techniques such as computer-synthesized speech (e.g., object name and direction), spatial audio, head rotation, and modulated audio (pitch and amplitude modulation), past works have found success in using auditory sensory augmentation to provide users with accurate spatial information of their surrounding environment with very little training [15, 19–22].”
From An investigation into the effectiveness of using acoustic touch to assist people who are blind (2023) · cites this paper as [20]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“The participants' feedback or task performance is used as an indicator to assess the efficiency of the conversion method in the conventional assessment [17], [40].”
From Deep Learning-Based Optimization of Visual–Auditory Sensory Substitution (2023) · cites this paper as [17]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Quattoni, A., & Torralba, A. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206537 · Cited by 1,463
Vision substitution by tactile image projection
BACH-Y-RITA, P., COLLINS, C. C., SAUNDERS, F. A., WHITE, B., & SCADDEN, L. (1969) · Nature · doi:10.1038/221963a0 · Cited by 956 · cited 2×
“stimulators which vibrate against the skin of the back conveying shapes as captured from a television camera (Bach-y-Rita et al., 1969); and the “Tongue Display Unit” (TDU), generating electro-tactile pulses displayed on the tongue (Kaczmarek, 2011).”
An experimental system for auditory image representations
Meijer, P. B. L. (1992) · IEEE Transactions on Biomedical Engineering · doi:10.1109/10.121642 · Cited by 900 · cited 5×
“In the visual-to-auditory domain, “The vOICe” introduced the sweep-line technique that generated auditory representations, called soundscapes, which map the xaxis of captured images to the time domain and the y-axis to the frequency domain (Meijer, 1992).”
InfluentialDecomposing a scene into geometric and semantically consistent regions
Gould, S., Fulton, R., & Koller, D. (2009) · 2009 IEEE 12th International Conference on Computer Vision · doi:10.1109/iccv.2009.5459211 · Cited by 672 · cited 1×
“We designed an online task with 30 stimuli consisting of 3 categories: indoor scenes (Adapted from Quattoni et al., 2009), outdoor scenes (Adapted from Gould et al., 2009) and simple shapes that we designed.”
Tactile Acuity is Enhanced in Blindness
Goldreich, D., & Kanics, I. M. (2003) · The Journal of Neuroscience · doi:10.1523/jneurosci.23-08-03439.2003 · Cited by 427 · cited 1×
“These devices rely on the improved auditory and tactile sensations of the blind (For examples see, Goldreich et al., 2003; Gougoux et al., 2004; Sathian et al., 2010) and aim to generate a shape-preserving representation of the environment as captured by an input device, e.g., a camera.”
Pitch discrimination in the early blind
Gougoux, F., Lepore, F., Lassonde, M., Voss, P., Zatorre, R. J., & Belin, P. (2004) · Nature · doi:10.1038/430309a · Cited by 422
Striem-Amit, E., Cohen, L., Dehaene, S., & Amedi, A. (2012) · Neuron · doi:10.1016/j.neuron.2012.08.026 · Cited by 294 · cited 3×
“In contrast, tasks that require higher resolution such as assessing facial expressions (Striem-Amit et al., 2012b) could benefit from an SSD with a higher resolution, such as the vOICe, which will have the upper hand.”
Self-citationAuvray, M., Hanneton, S., & O'Regan, J. K. (2007) · Perception · doi:10.1068/p5631 · Cited by 268 · cited 1×
“Various psychophysical experiments have demonstrated the ability of blind users to perform object and letter identification and discrimination (Auvray et al., 2007; Brown et al., 2011; Striem-Amit et al., 2012a), recognize patterns (Poirier et al., 2007), pair socks by color (Bologna et al., 2009),…”
Visual experiences in the blind induced by an auditory sensory substitution device
Ward, J., & Meijer, P. (2010) · Consciousness and Cognition · doi:10.1016/j.concog.2009.10.006 · Cited by 247
Quattoni, A., & Torralba, A. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvprw.2009.5206537 · Cited by 228
Cited by
Bhowmick, A., & Hazarika, S. M. (2017) · Journal on Multimodal User Interfaces · doi:10.1007/s12193-016-0235-6 · Cited by 271
“d EyeMusic: a sensory substitution device that conveys visual information via an auditory experience of musical notes, developed at the Hebrew University of Jerusalem ([1], photo credit: Maxim Dupliy, Amir Amedi and Shelly Levy-Tzedek).”
BackgroundOrigins of the specialization for letters and numbers in ventral occipitotemporal cortex
Hannagan, T., Amedi, A., Cohen, L., Dehaene-Lambertz, G., & Dehaene, S. (2015) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2015.05.006 · Cited by 233
Self-citationGori, M., Cappagli, G., Tonelli, A., Baud-Bovy, G., & Finocchietti, S. (2016) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2016.06.043 · Cited by 135
The “EyeCane”, a new electronic travel aid for the blind: Technology, behavior & swift learning
Maidenbaum, S., Hanassy, S., Abboud, S., Buchs, G., Chebat, D.-R., Levy-Tzedek, S., & Amedi, A. (2014) · Restorative Neurology and Neuroscience · doi:10.3233/rnn-130351 · Cited by 132
“See (Bach-y-Rita, 1972; Meijer, 1992; Capelle et al., 1998; Hanneton et al., 2010; Williams et al., 2011; Kärcher et al., 2012; LevyTzedek et al., 2012; Abboud et al., 2014) for examples of such devices and (Maidenbaum and Amedi, 2012; Maidenbaum, 2014) for recent reviews on the use of such devices for visual rehabili…”
Self-citationA number-form area in the blind
Abboud, S., Maidenbaum, S., Dehaene, S., & Amedi, A. (2015) · Nature Communications · doi:10.1038/ncomms7026 · Cited by 151
“We used the EyeMusic SSD to generate the auditory representation of the symbols I,V and X in White, Blue and Red, leading to nine stimuli in total.”
InfluentialSelf-citation
Metadata
- Authors
- Abboud, S., Hanassy, S., Levy-Tzedek, S., Maidenbaum, S., & Amedi, A.
- Affiliations
- Hebrew University of Jerusalem (IL)
- Department of Medical Neurobiology, The Institute for Medical Research Israel-Canada, Faculty of Medicine, The Hebrew University of Jerusalem, Jerusalem, Israel
- Corresponding author
- Amedi, Amir
- Venue
- Restorative Neurology and Neuroscience
- Publisher
- SAGE Publications
- Published
- 2014-01-01
- Volume
- 32
- Issue
- 2
- Pages
- 247-257
- ISSN
- 0922-6028 (ISSN-L)
- 1878-3627
- OpenAlex
- W1922152898
- Cited by
- 199
- References
- 43
- Citations by year
- 2026: 11 · 2025: 14 · 2024: 16 · 2023: 17 · 2022: 20 · 2021: 16 · 2020: 18 · 2019: 17 · 2018: 10 · 2017: 12 · 2016: 17 · 2015: 22 · 2014: 9
- FWCI
- 11.15
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Restorative Neurology and Neuroscience (publishedVersion, cc-by-nc)
- PubMed (publishedVersion)
- Funders
- James S. McDonnell Foundation (220020284)
- Israel Science Foundation (ISF 1684/08, 1684/08)
- FP7 Ideas: European Research Council (310809)
- Topics
- Life Sciences › Neuroscience › Cognitive Neuroscience › Tactile and Sensory Interactions
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Multisensory perception and integration
- Health Sciences › Health Professions › Occupational Therapy › Assistive Technology in Communication and Mobility
- Keywords
- sensory substitution devices, color perception, shape perception, blind users, pentatonic scale, visual rehabilitation
- MeSH terms
- Acoustic Stimulation, Adult, Auditory Perception, Blindness, Female, Humans, Male, Middle Aged, Neuronal Plasticity, Sensation, Persons with Visual Disabilities, Education of Persons with Visual Disabilities
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: EyeMusic: Introducing a “visual” colorful experience for the blind using auditory sensory substitution. https://app.cochat.ai/papers/10.3233/rnn-130338
- The paper
- Abboud, S., Hanassy, S., Levy-Tzedek, S., Maidenbaum, S., & Amedi, A. (2014). EyeMusic: Introducing a “visual” colorful experience for the blind using auditory sensory substitution. Restorative Neurology and Neuroscience. https://doi.org/10.3233/rnn-130338
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.