EyeMusic: Introducing a “visual” colorful experience for the blind using auditory sensory substitution
Open accessThe DOI matches its Crossref record; 42 of the 43 references it cites were found in a registry or an index.
Abstract
Purpose: Sensory-substitution devices (SSDs) provide auditory or tactile representations of visual information. These devices often generate unpleasant sensations and mostly lack color information. We present here a novel SSD aimed at addressing these issues. Methods: We developed the EyeMusic, a novel visual-to-auditory SSD for the blind, providing both shape and color information. Our design uses musical notes on a pentatonic scale generated by natural instruments to convey the visual information in a pleasant manner. A short behavioral protocol was utilized to train the blind to extract shape and color information, and test their acquired abilities. Finally, we conducted a survey and a comparison task to assess the pleasantness of the generated auditory stimuli. Results: We show that basic shape and color information can be decoded from the generated auditory stimuli. High performance levels were achieved by all participants following as little as 2–3 hours of training. Furthermore, we show that users indeed found the stimuli pleasant and potentially tolerable for prolonged use. Conclusions: The novel EyeMusic algorithm provides an intuitive and relatively pleasant way for the blind to extract shape and color information. We suggest that this might help facilitating visual rehabilitation because of the added functionality and enhanced pleasantness.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Using techniques such as computer-synthesized speech (e.g., object name and direction), spatial audio, head rotation, and modulated audio (pitch and amplitude modulation), past works have found success in using auditory sensory augmentation to provide users with accurate spatial information of their surrounding environment with very little training [15, 19–22].”
From An investigation into the effectiveness of using acoustic touch to assist people who are blind (2023) · cites this paper as [20]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“The participants' feedback or task performance is used as an indicator to assess the efficiency of the conversion method in the conventional assessment [17], [40].”
From Deep Learning-Based Optimization of Visual–Auditory Sensory Substitution (2023) · cites this paper as [17]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Quattoni, A., & Torralba, A. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206537 · Cited by 1,463
Vision substitution by tactile image projection
BACH-Y-RITA, P., COLLINS, C. C., SAUNDERS, F. A., WHITE, B., & SCADDEN, L. (1969) · Nature · doi:10.1038/221963a0 · Cited by 956 · cited 2×
“stimulators which vibrate against the skin of the back conveying shapes as captured from a television camera (Bach-y-Rita et al., 1969); and the “Tongue Display Unit” (TDU), generating electro-tactile pulses displayed on the tongue (Kaczmarek, 2011).”
An experimental system for auditory image representations
Meijer, P. B. L. (1992) · IEEE Transactions on Biomedical Engineering · doi:10.1109/10.121642 · Cited by 900 · cited 5×
“In the visual-to-auditory domain, “The vOICe” introduced the sweep-line technique that generated auditory representations, called soundscapes, which map the xaxis of captured images to the time domain and the y-axis to the frequency domain (Meijer, 1992).”
InfluentialDecomposing a scene into geometric and semantically consistent regions
Gould, S., Fulton, R., & Koller, D. (2009) · 2009 IEEE 12th International Conference on Computer Vision · doi:10.1109/iccv.2009.5459211 · Cited by 672 · cited 1×
“We designed an online task with 30 stimuli consisting of 3 categories: indoor scenes (Adapted from Quattoni et al., 2009), outdoor scenes (Adapted from Gould et al., 2009) and simple shapes that we designed.”
Tactile Acuity is Enhanced in Blindness
Goldreich, D., & Kanics, I. M. (2003) · The Journal of Neuroscience · doi:10.1523/jneurosci.23-08-03439.2003 · Cited by 427 · cited 1×
“These devices rely on the improved auditory and tactile sensations of the blind (For examples see, Goldreich et al., 2003; Gougoux et al., 2004; Sathian et al., 2010) and aim to generate a shape-preserving representation of the environment as captured by an input device, e.g., a camera.”
Pitch discrimination in the early blind
Gougoux, F., Lepore, F., Lassonde, M., Voss, P., Zatorre, R. J., & Belin, P. (2004) · Nature · doi:10.1038/430309a · Cited by 422
Striem-Amit, E., Cohen, L., Dehaene, S., & Amedi, A. (2012) · Neuron · doi:10.1016/j.neuron.2012.08.026 · Cited by 294 · cited 3×
“In contrast, tasks that require higher resolution such as assessing facial expressions (Striem-Amit et al., 2012b) could benefit from an SSD with a higher resolution, such as the vOICe, which will have the upper hand.”
Self-citationAuvray, M., Hanneton, S., & O'Regan, J. K. (2007) · Perception · doi:10.1068/p5631 · Cited by 268 · cited 1×
“Various psychophysical experiments have demonstrated the ability of blind users to perform object and letter identification and discrimination (Auvray et al., 2007; Brown et al., 2011; Striem-Amit et al., 2012a), recognize patterns (Poirier et al., 2007), pair socks by color (Bologna et al., 2009),…”
Visual experiences in the blind induced by an auditory sensory substitution device
Ward, J., & Meijer, P. (2010) · Consciousness and Cognition · doi:10.1016/j.concog.2009.10.006 · Cited by 247
Quattoni, A., & Torralba, A. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvprw.2009.5206537 · Cited by 228
Capelle, C., Trullemans, C., Arno, P., & Veraart, C. (1998) · IEEE Transactions on Biomedical Engineering · doi:10.1109/10.720206 · Cited by 210 · cited 2×
“The “Prosthesis Substituting Vision by Audition” (PSVA), on the other hand, used an auditory frequency for each spatial location and generated the difference between each two subsequent images relying on camera movement by the user (Capelle et al., 1998).”
The role of color information on object recognition: A review and meta-analysis
Bramão, I., Reis, A., Petersson, K. M., & Faísca, L. (2011) · Acta Psychologica · doi:10.1016/j.actpsy.2011.06.010 · Cited by 193 · cited 2×
“However, color information is an important feature in vision for tasks such as object recognition and scene segmentation (Bramão et al., 2011; Goffaux et al., 2005; Yip et al., 2002), even if possible to perform without it.”
Neural correlates of virtual route reCognition in congenital blindness
Kupers, R., Chebat, D. R., Madsen, K. H., Paulson, O. B., & Ptito, M. (2010) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1006199107 · Cited by 187 · cited 2×
“, 2010), navigate, and avoid obstacles (Chebat et al., 2011; Kupers et al., 2010) and even perform complex tasks such as recognizing facial expressions (Striem-Amit et al.”
Contribution of color to face recognition
Yip, A. W., & Sinha, P. (2002) · Perception · doi:10.1068/p3376 · Cited by 161
Perception With Compensatory Devices: From Sensory Substitution to Sensorimotor Extension
Auvray, M., & Myin, E. (2009) · Cognitive Science · doi:10.1111/j.1551-6709.2009.01040.x · Cited by 155 · cited 2×
“Several attempts at converting visual information to auditory and tactile cues have been made in the past creating a family of Sensory Substitution Devices (SSDs) which enable shape and scene perception by the blind (Reviewed in Auvray et al., 2009).”
Pleasant music overcomes the loss of awareness in patients with visual neglect
Soto, D., Funes, M. J., Guzmán-García, A., Warbrick, T., Rotshtein, P., & Humphreys, G. W. (2009) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.0811681106 · Cited by 141 · cited 2×
“Previous studies investigated the role of pleasant music when treating patients with unilateral neglect after stroke (Chen et al., 2013; Sarkamo et al., 2012; Soto et al., 2009; Tsai et al., 2013).”
Navigation with a sensory substitution device in congenitally blind individuals
Chebat, D.-R., Schneider, F. C., Kupers, R., & Ptito, M. (2011) · Neuroreport · doi:10.1097/wnr.0b013e3283462def · Cited by 132 · cited 2×
“, 2010), navigate, and avoid obstacles (Chebat et al., 2011; Kupers et al., 2010) and even perform complex tasks such as recognizing facial expressions (Striem-Amit et al.”
Music listening after stroke: beneficial effects and potential neural mechanisms
Särkämö, T., & Soto, D. (2012) · Annals of the New York Academy of Sciences · doi:10.1111/j.1749-6632.2011.06405.x · Cited by 131
‘Visual’ Acuity of the Congenitally Blind Using Visual-to-Auditory Sensory Substitution
Striem-Amit, E., Guendelman, M., & Amedi, A. (2012) · PLoS ONE · doi:10.1371/journal.pone.0033136 · Cited by 117 · cited 3×
“In contrast, tasks that require higher resolution such as assessing facial expressions (Striem-Amit et al., 2012b) could benefit from an SSD with a higher resolution, such as the vOICe, which will have the upper hand.”
Self-citationCross-modal plasticity of tactile perception in blindness
Sathian, K., & Stilla, R. (2010) · Restorative Neurology and Neuroscience · doi:10.3233/rnn-2010-0534 · Cited by 115
Goffaux, V., Jacques, C., Mouraux, A., Oliva, A., Schyns, P. G., & Rossion, B. (2005) · Visual Cognition · doi:10.1080/13506280444000562 · Cited by 113 · cited 1×
“However, color information is an important feature in vision for tasks such as object recognition and scene segmentation (Bramão et al., 2011; Goffaux et al., 2005; Yip et al., 2002), even if possible to perform without it.”
The tongue display unit (TDU) for electrotactile spatiotemporal pattern presentation
Kaczmarek, K. A. (2011) · Scientia Iranica · doi:10.1016/j.scient.2011.08.020 · Cited by 103 · cited 2×
“, 1969); and the “Tongue Display Unit” (TDU), generating electro-tactile pulses displayed on the tongue (Kaczmarek, 2011).”
Mapping unpleasantness of sounds to their auditory representation
Kumar, S., Forster, H. M., Bailey, P., & Griffiths, T. D. (2008) · The Journal of the Acoustical Society of America · doi:10.1121/1.3006380 · Cited by 94 · cited 2×
“This was done because sounds with high energy in the 2500-5000 Hz range have been linked with unpleasantness (Kumar et al., 2008).”
Kasten, E., Poggel, D. A., Müller‐Oehring, E. M., Gothe, J., Schulte, T., & Sabel, B. (1999) · Restorative Neurology and Neuroscience · doi:10.3233/rnn-1999-00123 · Cited by 86 · cited 1×
“In the future, the EyeMusic can be used together with other visual restoration approaches (for examples of such approaches see Dagnelie 2012; Reich et al., 2012; Kasten et al., 1999) such as the evolving visual prostheses.”
Proulx, M. J., Stoerig, P., Ludowig, E., & Knoll, I. (2008) · PLoS ONE · doi:10.1371/journal.pone.0001840 · Cited by 81 · cited 2×
“, 2009), localize visual stimuli (Proulx et al., 2008), reach for targets (Levy-Tzedek et al.”
Fast, accurate reaching movements with a visual-to-auditory sensory substitution device
Levy-Tzedek, S., Hanassy, S., Abboud, S., Maidenbaum, S., & Amedi, A. (2012) · Restorative Neurology and Neuroscience · doi:10.3233/rnn-2012-110219 · Cited by 79 · cited 4×
“Users are expected to perform better in tasks such involving color-driven reaching (Levy-Tzedek et al., 2012a) and figure-ground segregation compared with veteran SSDs that do not provide color information.”
InfluentialSelf-citationReich, L., Maidenbaum, S., & Amedi, A. (2012) · Current Opinion in Neurology · doi:10.1097/wco.0b013e32834ed723 · Cited by 78 · cited 4×
“, 2012b) using SSDs, proving their feasibility (Detailed review in Auvray et al., 2009; Reich et al., 2012) and leading to the conclusion that SSDs can and should be adopted for rehabilitation either as standalone devices or as part of combined solutions addressing the various needs of the blind.”
InfluentialSelf-citationPleasant music improves visual attention in patients with unilateral neglect after stroke
Chen, M.-C., Tsai, P.-L., Huang, Y., & Lin, K. (2012) · Brain Injury · doi:10.3109/02699052.2012.722255 · Cited by 76 · cited 3×
“Previous studies investigated the role of pleasant music when treating patients with unilateral neglect after stroke (Chen et al., 2013; Sarkamo et al., 2012; Soto et al., 2009; Tsai et al., 2013).”
Brown, D., Macpherson, T., & Ward, J. (2011) · Perception · doi:10.1068/p6952 · Cited by 72 · cited 2×
“Various psychophysical experiments have demonstrated the ability of blind users to perform object and letter identification and discrimination (Auvray et al., 2007; Brown et al., 2011; Striem-Amit et al., 2012a), recognize patterns (Poirier et al.”
Retinal implants: emergence of a multidisciplinary field
Dagnelie, G. (2012) · Current Opinion in Neurology · doi:10.1097/wco.0b013e32834f02c3 · Cited by 71 · cited 1×
“In the future, the EyeMusic can be used together with other visual restoration approaches (for examples of such approaches see Dagnelie 2012; Reich et al., 2012; Kasten et al., 1999) such as the evolving visual prostheses.”
Bioinspired Applications in Artificial and Natural Computation
Hutchison, D., Naor, M., Mitchell, J. C., Ferrández, J. M., Mattern, F., Weikum, G., Tygar, D., Sudan, M., Vardi, M. Y., Takeo Kanade, A., et al. (2009) · Lecture notes in computer science · doi:10.1007/978-3-642-02267-8 · Cited by 55
Listening to Classical Music Ameliorates Unilateral Neglect After Stroke
Tsai, P.-L., Chen, M.-C., Huang, Y., Lin, K., Chen, K., & Hsu, Y.-W. (2013) · American Journal of Occupational Therapy · doi:10.5014/ajot.2013.006312 · Cited by 54 · cited 2×
“Previous studies investigated the role of pleasant music when treating patients with unilateral neglect after stroke (Chen et al., 2013; Sarkamo et al., 2012; Soto et al., 2009; Tsai et al., 2013).”
Renier, L., & De Volder, A. G. (2010) · Disability and Rehabilitation: Assistive Technology · doi:10.3109/17483100903253936 · Cited by 51 · cited 1×
“…2009), localize visual stimuli (Proulx et al., 2008), reach for targets (Levy-Tzedek et al., 2012a; Levy-Tzedek et al., 2012b), extract depth cues (Renier et al., 2010), navigate, and avoid obstacles (Chebat et al., 2011; Kupers et al., 2010) and even perform complex tasks such as recognizing…”
Levy-Tzedek, S., Novick, I., Arbel, R., Abboud, S., Maidenbaum, S., Vaadia, E., & Amedi, A. (2012) · Scientific Reports · doi:10.1038/srep00949 · Cited by 51 · cited 4×
“Users are expected to perform better in tasks such involving color-driven reaching (Levy-Tzedek et al., 2012a) and figure-ground segregation compared with veteran SSDs that do not provide color information.”
InfluentialSelf-citationPattern recognition using a device substituting audition for vision in blindfolded sighted subjects
Poirier, C., De Volder, A., Tranduy, D., & Scheiber, C. (2007) · Neuropsychologia · doi:10.1016/j.neuropsychologia.2006.09.018 · Cited by 46 · cited 1×
“…object and letter identification and discrimination (Auvray et al., 2007; Brown et al., 2011; Striem-Amit et al., 2012a), recognize patterns (Poirier et al., 2007), pair socks by color (Bologna et al., 2009), localize visual stimuli (Proulx et al., 2008), reach for targets (Levy-Tzedek et…”
Transforming 3D Coloured Pixels into Musical Instrument Notes for Vision Substitution Applications
Bologna, G., Deville, B., Pun, T., & Vinckenbosch, M. (2007) · EURASIP Journal on Image and Video Processing · doi:10.1155/2007/76204 · Cited by 42 · cited 3×
“SeeColOr, for example, used timbre information to represent color hues, pitch to represent saturation and an addition of two extra timbres, simultaneous to the hue timbres, to represent luminance.”
SOUNDVIEW: SENSING COLOR IMAGES BY KINESTHETIC AUDIO
van den Doel, K. (2003) · Cited by 36
Steeves, J. K. E., & Harris, L. R. (2012) · Cambridge University Press eBooks · doi:10.1017/cbo9781139136907.001 · Cited by 33
Learning and perceiving colors haptically
Kahol, K., French, J., Bratton, L., & Panchanathan, S. (2006) · International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS) · doi:10.1145/1168987.1169017 · Cited by 26 · cited 2×
“Some conveyed color as a single point, e.g., Soundview (Doel, 2003), Haptic-Colour Glove (Kahol et al., 2006), while others conveyed horizontal lines, e.g., SeeColOr (Bologna et al., 2007).”
Wright, T., & Ward, J. (2013) · Quarterly Journal of Experimental Psychology · doi:10.1080/17470218.2012.754911 · Cited by 20
Collignon, O., Dormal, G., & Leporé, F. (2012) · Cambridge University Press eBooks · doi:10.1017/cbo9781139136907 · Cited by 15
‘Visual’ acuity of the congenitally blind using visual-to-auditory sensory substitution
Striem-Amit, E., Guendelman, M., & Amedi, A. (2012) · Seeing and Perceiving · doi:10.1163/187847612x648206 · Cited by 6
Self-citation
Cited by
Bhowmick, A., & Hazarika, S. M. (2017) · Journal on Multimodal User Interfaces · doi:10.1007/s12193-016-0235-6 · Cited by 271
“d EyeMusic: a sensory substitution device that conveys visual information via an auditory experience of musical notes, developed at the Hebrew University of Jerusalem ([1], photo credit: Maxim Dupliy, Amir Amedi and Shelly Levy-Tzedek).”
BackgroundOrigins of the specialization for letters and numbers in ventral occipitotemporal cortex
Hannagan, T., Amedi, A., Cohen, L., Dehaene-Lambertz, G., & Dehaene, S. (2015) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2015.05.006 · Cited by 233
Self-citationGori, M., Cappagli, G., Tonelli, A., Baud-Bovy, G., & Finocchietti, S. (2016) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2016.06.043 · Cited by 135
The “EyeCane”, a new electronic travel aid for the blind: Technology, behavior & swift learning
Maidenbaum, S., Hanassy, S., Abboud, S., Buchs, G., Chebat, D.-R., Levy-Tzedek, S., & Amedi, A. (2014) · Restorative Neurology and Neuroscience · doi:10.3233/rnn-130351 · Cited by 132
“See (Bach-y-Rita, 1972; Meijer, 1992; Capelle et al., 1998; Hanneton et al., 2010; Williams et al., 2011; Kärcher et al., 2012; LevyTzedek et al., 2012; Abboud et al., 2014) for examples of such devices and (Maidenbaum and Amedi, 2012; Maidenbaum, 2014) for recent reviews on the use of such devices for visual rehabili…”
Self-citationA number-form area in the blind
Abboud, S., Maidenbaum, S., Dehaene, S., & Amedi, A. (2015) · Nature Communications · doi:10.1038/ncomms7026 · Cited by 151
“We used the EyeMusic SSD to generate the auditory representation of the symbols I,V and X in White, Blue and Red, leading to nine stimuli in total.”
InfluentialSelf-citation
Metadata
- Authors
- Abboud, S., Hanassy, S., Levy-Tzedek, S., Maidenbaum, S., & Amedi, A.
- Affiliations
- Hebrew University of Jerusalem (IL)
- Department of Medical Neurobiology, The Institute for Medical Research Israel-Canada, Faculty of Medicine, The Hebrew University of Jerusalem, Jerusalem, Israel
- Corresponding author
- Amedi, Amir
- Venue
- Restorative Neurology and Neuroscience
- Publisher
- SAGE Publications
- Published
- 2014-01-01
- Volume
- 32
- Issue
- 2
- Pages
- 247-257
- ISSN
- 0922-6028 (ISSN-L)
- 1878-3627
- OpenAlex
- W1922152898
- Cited by
- 199
- References
- 43
- Citations by year
- 2026: 11 · 2025: 14 · 2024: 16 · 2023: 17 · 2022: 20 · 2021: 16 · 2020: 18 · 2019: 17 · 2018: 10 · 2017: 12 · 2016: 17 · 2015: 22 · 2014: 9
- FWCI
- 11.15
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Restorative Neurology and Neuroscience (publishedVersion, cc-by-nc)
- PubMed (publishedVersion)
- Funders
- James S. McDonnell Foundation (220020284)
- Israel Science Foundation (ISF 1684/08, 1684/08)
- FP7 Ideas: European Research Council (310809)
- Topics
- Life Sciences › Neuroscience › Cognitive Neuroscience › Tactile and Sensory Interactions
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Multisensory perception and integration
- Health Sciences › Health Professions › Occupational Therapy › Assistive Technology in Communication and Mobility
- Keywords
- sensory substitution devices, color perception, shape perception, blind users, pentatonic scale, visual rehabilitation
- MeSH terms
- Acoustic Stimulation, Adult, Auditory Perception, Blindness, Female, Humans, Male, Middle Aged, Neuronal Plasticity, Sensation, Persons with Visual Disabilities, Education of Persons with Visual Disabilities
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.