EyeMusic: Introducing a “visual” colorful experience for the blind using auditory sensory substitution
Open accessThe DOI matches its Crossref record; 42 of the 43 references it cites were found in a registry or an index.
Abstract
Purpose: Sensory-substitution devices (SSDs) provide auditory or tactile representations of visual information. These devices often generate unpleasant sensations and mostly lack color information. We present here a novel SSD aimed at addressing these issues. Methods: We developed the EyeMusic, a novel visual-to-auditory SSD for the blind, providing both shape and color information. Our design uses musical notes on a pentatonic scale generated by natural instruments to convey the visual information in a pleasant manner. A short behavioral protocol was utilized to train the blind to extract shape and color information, and test their acquired abilities. Finally, we conducted a survey and a comparison task to assess the pleasantness of the generated auditory stimuli. Results: We show that basic shape and color information can be decoded from the generated auditory stimuli. High performance levels were achieved by all participants following as little as 2–3 hours of training. Furthermore, we show that users indeed found the stimuli pleasant and potentially tolerable for prolonged use. Conclusions: The novel EyeMusic algorithm provides an intuitive and relatively pleasant way for the blind to extract shape and color information. We suggest that this might help facilitating visual rehabilitation because of the added functionality and enhanced pleasantness.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Using techniques such as computer-synthesized speech (e.g., object name and direction), spatial audio, head rotation, and modulated audio (pitch and amplitude modulation), past works have found success in using auditory sensory augmentation to provide users with accurate spatial information of their surrounding environment with very little training [15, 19–22].”
From An investigation into the effectiveness of using acoustic touch to assist people who are blind (2023) · cites this paper as [20]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“The participants' feedback or task performance is used as an indicator to assess the efficiency of the conversion method in the conventional assessment [17], [40].”
From Deep Learning-Based Optimization of Visual–Auditory Sensory Substitution (2023) · cites this paper as [17]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Quattoni, A., & Torralba, A. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206537 · Cited by 1,463
Vision substitution by tactile image projection
BACH-Y-RITA, P., COLLINS, C. C., SAUNDERS, F. A., WHITE, B., & SCADDEN, L. (1969) · Nature · doi:10.1038/221963a0 · Cited by 956 · cited 2×
“stimulators which vibrate against the skin of the back conveying shapes as captured from a television camera (Bach-y-Rita et al., 1969); and the “Tongue Display Unit” (TDU), generating electro-tactile pulses displayed on the tongue (Kaczmarek, 2011).”
An experimental system for auditory image representations
Meijer, P. B. L. (1992) · IEEE Transactions on Biomedical Engineering · doi:10.1109/10.121642 · Cited by 900 · cited 5×
“In the visual-to-auditory domain, “The vOICe” introduced the sweep-line technique that generated auditory representations, called soundscapes, which map the xaxis of captured images to the time domain and the y-axis to the frequency domain (Meijer, 1992).”
InfluentialDecomposing a scene into geometric and semantically consistent regions
Gould, S., Fulton, R., & Koller, D. (2009) · 2009 IEEE 12th International Conference on Computer Vision · doi:10.1109/iccv.2009.5459211 · Cited by 672 · cited 1×
“We designed an online task with 30 stimuli consisting of 3 categories: indoor scenes (Adapted from Quattoni et al., 2009), outdoor scenes (Adapted from Gould et al., 2009) and simple shapes that we designed.”
Tactile Acuity is Enhanced in Blindness
Goldreich, D., & Kanics, I. M. (2003) · The Journal of Neuroscience · doi:10.1523/jneurosci.23-08-03439.2003 · Cited by 427 · cited 1×
“These devices rely on the improved auditory and tactile sensations of the blind (For examples see, Goldreich et al., 2003; Gougoux et al., 2004; Sathian et al., 2010) and aim to generate a shape-preserving representation of the environment as captured by an input device, e.g., a camera.”
Pitch discrimination in the early blind
Gougoux, F., Lepore, F., Lassonde, M., Voss, P., Zatorre, R. J., & Belin, P. (2004) · Nature · doi:10.1038/430309a · Cited by 422
Striem-Amit, E., Cohen, L., Dehaene, S., & Amedi, A. (2012) · Neuron · doi:10.1016/j.neuron.2012.08.026 · Cited by 294 · cited 3×
“In contrast, tasks that require higher resolution such as assessing facial expressions (Striem-Amit et al., 2012b) could benefit from an SSD with a higher resolution, such as the vOICe, which will have the upper hand.”
Self-citationAuvray, M., Hanneton, S., & O'Regan, J. K. (2007) · Perception · doi:10.1068/p5631 · Cited by 268 · cited 1×
“Various psychophysical experiments have demonstrated the ability of blind users to perform object and letter identification and discrimination (Auvray et al., 2007; Brown et al., 2011; Striem-Amit et al., 2012a), recognize patterns (Poirier et al., 2007), pair socks by color (Bologna et al., 2009),…”
Visual experiences in the blind induced by an auditory sensory substitution device
Ward, J., & Meijer, P. (2010) · Consciousness and Cognition · doi:10.1016/j.concog.2009.10.006 · Cited by 247
Quattoni, A., & Torralba, A. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvprw.2009.5206537 · Cited by 228
Cited by
Bhowmick, A., & Hazarika, S. M. (2017) · Journal on Multimodal User Interfaces · doi:10.1007/s12193-016-0235-6 · Cited by 271
“d EyeMusic: a sensory substitution device that conveys visual information via an auditory experience of musical notes, developed at the Hebrew University of Jerusalem ([1], photo credit: Maxim Dupliy, Amir Amedi and Shelly Levy-Tzedek).”
BackgroundOrigins of the specialization for letters and numbers in ventral occipitotemporal cortex
Hannagan, T., Amedi, A., Cohen, L., Dehaene-Lambertz, G., & Dehaene, S. (2015) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2015.05.006 · Cited by 233
Self-citationGori, M., Cappagli, G., Tonelli, A., Baud-Bovy, G., & Finocchietti, S. (2016) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2016.06.043 · Cited by 135
The “EyeCane”, a new electronic travel aid for the blind: Technology, behavior & swift learning
Maidenbaum, S., Hanassy, S., Abboud, S., Buchs, G., Chebat, D.-R., Levy-Tzedek, S., & Amedi, A. (2014) · Restorative Neurology and Neuroscience · doi:10.3233/rnn-130351 · Cited by 132
“See (Bach-y-Rita, 1972; Meijer, 1992; Capelle et al., 1998; Hanneton et al., 2010; Williams et al., 2011; Kärcher et al., 2012; LevyTzedek et al., 2012; Abboud et al., 2014) for examples of such devices and (Maidenbaum and Amedi, 2012; Maidenbaum, 2014) for recent reviews on the use of such devices for visual rehabili…”
Self-citationA number-form area in the blind
Abboud, S., Maidenbaum, S., Dehaene, S., & Amedi, A. (2015) · Nature Communications · doi:10.1038/ncomms7026 · Cited by 151
“We used the EyeMusic SSD to generate the auditory representation of the symbols I,V and X in White, Blue and Red, leading to nine stimuli in total.”
InfluentialSelf-citationDesigning sensory-substitution devices: Principles, pitfalls and potential1
Kristjánsson, Á., Moldoveanu, A., Jóhannesson, Ó. I., Balan, O., Spagnol, S., Valgeirsdóttir, V. V., & Unnthorsson, R. (2016) · Restorative Neurology and Neuroscience · doi:10.3233/rnn-160647 · Cited by 116
“In the image-to-sound conversion algorithm of EyeMusic (Abboud et al., 2014) higher musical notes represent pixels located higher on the y-axis, the delay of sound presentation corresponds to the location of a pixel on the x-axis (a larger delay for the sounds from the right and a smaller one for the sounds from the l…”
BackgroundInfluentialNavigation using sensory substitution in real and virtual mazes
Chebat, D.-R., Maidenbaum, S., & Amedi, A. (2015) · PLOS ONE · doi:10.1371/journal.pone.0126307 · Cited by 99
“Are differences in the ability to navigate between blind and sighted people simply based on a lack of current visual information? Would it be possible to improve the performance of the blind in a navigation task to the point where it would be similar to the performance of the sighted? Or does congenital blindness prev…”
BackgroundSelf-citationIs there a future for sensory substitution outside academic laboratories?
Elli, G. V., Benetti, S., & Collignon, O. (2014) · Multisensory Research · doi:10.1163/22134808-00002460 · Cited by 90
“Since the seminal work of Paul Bach-y-Rita, who created the first SSD translating visual information into tactile input (Bach-y-Rita et al. , 1969), a number of visual-to-tactile (e.g. TDU, Bach-y-Rita et al. , 1998) and visual-to-auditory (PSVA, Capelle et al. , 1992, 1998; vOICe, Meijer, 1992; VIBE, Auvray et al. , 2…”
Computer Vision for the Visually Impaired: the Sound of Vision System
Caraiman, S., Morar, A., Owczarek, M., Burlacu, A., Rzeszotarski, D., Botezatu, N., Herghelegiu, P., Moldoveanu, F., Strumillo, P., & Moldoveanu, A. (2017) · 2017 IEEE International Conference on Computer Vision Workshops (ICCVW) · doi:10.1109/iccvw.2017.175 · Cited by 85
“Sensory substitution devices are designed to convey visual information to the visually impaired by substituting visual information into one of their intact senses [1, 3, 5, 6, 15, 30, 35, 42].”
BackgroundKönig, S. U., Schumann, F., Keyser, J., Goeke, C., Carina, K., Susan, W., Lytochkin, A., Manuel, E., Brunsch, V., Wahn, B., et al. (2016) · PLoS ONE · doi:10.1371/journal.pone.0166647 · Cited by 85
“Since then, these results were supported by numerous experiments using vision-to-tactile substitution [17], or other sensory substitution devices like vision-to-auditory substitution [18,19], and vestibular-to-tactile substitution [20].”
ResultVisual rehabilitation: visual scanning, multisensory stimulation and vision restoration trainings
Dundon, N. M., Bertini, C., Làdavas, E., Sabel, B. A., & Gall, C. (2015) · Frontiers in Behavioral Neuroscience · doi:10.3389/fnbeh.2015.00192 · Cited by 88
Cecchetti, L., Kupers, R., Ptito, M., Pietrini, P., & Ricciardi, E. (2016) · Frontiers in Systems Neuroscience · doi:10.3389/fnsys.2016.00089 · Cited by 70
“Blind individuals can even process shape and color features by means of SSD-generated auditory stimuli (Abboud et al., 2014).”
BackgroundMethodInfluentialMultisensory inclusive design with sensory substitution
Esenkaya, T., Lloyd‐Esenkaya, V., O'Neill, E. J., & Proulx, M. J. (2020) · Cognitive Research Principles and Implications · doi:10.1186/s41235-020-00240-7 · Cited by 94
Heimler, B., & Amedi, A. (2020) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2020.06.034 · Cited by 78
“Specifically, visual SSD algorithms convey whole-visual scene information through manipulations of specific auditory or tactile features, ultimately successfully conveying the shapes and locations of objects (Bach-y-Rita, 2004; Meijer, 1992a,b) and even their colors (Abboud et al., 2014).”
BackgroundSelf-citationNew Vision for Visual Prostheses
Farnum, A., & Pelled, G. (2020) · Frontiers in Neuroscience · doi:10.3389/fnins.2020.00036 · Cited by 58
Cieśla, K., Wolak, T. P., Lorens, A., Mentzel, M., Skarżyńśki, H., & Amedi, A. (2022) · Scientific Reports · doi:10.1038/s41598-022-06855-8 · Cited by 59
Self-citationObject detection and identification for blind people in video scene
Jabnoun, H., Benzarti, F., & Amiri, H. (2015) · International Conference on Intelligent Systems Design and Applications (ISDA) · doi:10.1109/isda.2015.7489256 · Cited by 55
“In this paper, we present an overview of vision substitution modalities [1] [2] [3] [4] [5] [6] [7] [8] [9] [10] [11] [12] and their functionalities.”
BackgroundMassiceti, D., Hicks, S. L., & van Rheede, J. J. (2018) · PLOS ONE · doi:10.1371/journal.pone.0199389 · Cited by 54
Hearing colors: an example of brain plasticity
Alfaro, A., Bernabéu, Á., Mateo, C. M., Parra, J., & Fernández, E. (2015) · Frontiers in Systems Neuroscience · doi:10.3389/fnsys.2015.00056 · Cited by 58
“Although, the most common modality to replace sight is touch, several authors proposed special devices for visual substitution by the auditory pathway in the context of real time reactivity (Meijer, 1992; Capelle et al., 1998; CronlyDillon et al., 1999; Striem-Amit et al., 2012; Abboud et al., 2014; Maidenbaum et al.,…”
BackgroundMethodKilian, J., Neugebauer, A., Scherffig, L., & Wahl, S. (2022) · Sensors · doi:10.3390/s22051859 · Cited by 59
“Many influential publications followed this example and developed SSDs for the tactile modality [34–37], one even suggesting using a glove as tactile interface [38]; others addressed further modalities, such as the auditory with so-called Auditory-Vision Substitution Systems (AVSS) [39–43].”
BackgroundSpatial Competence and Brain Plasticity in Congenital Blindness via Sensory Substitution Devices
Chebat, D., Schneider, F., & Ptito, M. (2020) · Frontiers in Neuroscience · doi:10.3389/fnins.2020.00815 · Cited by 61
Brain-Machine Interfaces to Assist the Blind
Ptito, M., Bleau, M., Djerourou, I., Paré, S., Schneider, F. C., & Chebat, D.-R. (2021) · Frontiers in Human Neuroscience · doi:10.3389/fnhum.2021.638887 · Cited by 45
Sensory translation between audition and vision
Spence, C. J., & Stefano, N. D. (2023) · Psychonomic Bulletin & Review · doi:10.3758/s13423-023-02343-w · Cited by 46
“…those scientists trying to develop more intuitive sensory substitution devices (to compensate for sensory loss, e.g., in blind or deaf people, see Abboud et al., 2014; Hamil-ton-Fletcher et al., 2016; Marks, 1983; Spence, 2018, as well as for [sensory] augmentation, e.g., Akinbiyi et al., 2006;…”
Navigation aid for blind persons by visual-to-auditory sensory substitution: A pilot study
Neugebauer, A., Rifai, K., Getzlaff, M., & Wahl, S. (2020) · PLOS ONE · doi:10.1371/journal.pone.0237344 · Cited by 41
“Two of the most popular devices in that field are the visual-to-auditory SSDs ‘the voice’ [24, 29–31] and ‘EyeMusic’ [32, 33].”
BackgroundMethodInfluentialCieśla, K., Wolak, T. P., Lorens, A., Heimler, B., Skarżyńśki, H., & Amedi, A. (2019) · Restorative Neurology and Neuroscience · doi:10.3233/rnn-190898 · Cited by 48
“…possible reason might be that the aforementioned SSD solutions use quite complex algorithms to convey a variety of visual features simultaneously, such as shape, location and even colors of objects (Striem-Amit et al., 2012; Maidenbaum et al., 2014; Levy-Tzedek et al., 2014; Abboud et al., 2014).”
BackgroundSelf-citationCross-Modal Correspondences Enhance Performance on a Colour-to-Sound Sensory Substitution Device
Hamilton-Fletcher, G., Wright, T. D., & Ward, J. (2016) · Multisensory Research · doi:10.1163/22134808-00002519 · Cited by 38
"I Always Wanted to See the Night Sky": Blind User Preferences for Sensory Substitution Devices
Hamilton-Fletcher, G., Obrist, M., Watten, P., Mengucci, M., & Ward, J. (2016) · Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems · doi:10.1145/2858036.2858241 · Cited by 36
“Only aesthetic ratings of two devices have been done previously by BVIPs [3].”
BackgroundBuimer, H. P., Bittner, M., Kostelijk, T., van der Geest, T. M., Nemri, A., Wezel, R. J. A. V., & Zhao, Y. (2018) · PLoS ONE · doi:10.1371/journal.pone.0194737 · Cited by 44
How Do We Experience Crossmodal Correspondent Mulsemedia Content?
Covaci, A., Saleme, E. B., Mesfin, G., Hussain, N., Kani‐Zabihi, E., & Ghinea, G. (2019) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2019.2941274 · Cited by 40
“, blind users [31], [32]); d) improving content recall through multisensory congruency in a variety of mulsemedia applications [33].”
Background(2016) · Symposium On Usable Privacy and Security · Cited by 35
Bermejo, F. R., Paolo, E. A. D., Hüg, M. X., & Arias, C. J. (2015) · Frontiers in Psychology · doi:10.3389/fpsyg.2015.00679 · Cited by 43
“This is consistent with previous results: it is possible, using only sound stimuli coupled to self-generated movements, to perform typical visual tasks (Proulx et al., 2008; Abboud et al., 2013; Viaud-Delmon and Warusfel, 2014).”
ResultPerception of Graphical Virtual Environments by Blind Users via Sensory Substitution
Maidenbaum, S., Buchs, G., Abboud, S., Lavi‐Rotbain, O., & Amedi, A. (2016) · PLoS ONE · doi:10.1371/journal.pone.0147501 · Cited by 40
Self-citationThe role of visual deprivation and experience on the performance of sensory substitution devices
Stronks, H. C., Nau, A. C., Ibbotson, M. R., & Barnes, N. (2015) · Brain Research · doi:10.1016/j.brainres.2015.06.033 · Cited by 31
“This “image-sweep line” technique has subsequently been deployed in the EyeMusic device for use in sensory substitution (Abboud et al., 2014).”
MethodBrown, D. J., & Proulx, M. J. (2016) · IEEE Journal of Selected Topics in Signal Processing · doi:10.1109/jstsp.2016.2543678 · Cited by 31
“In general the substituted modality is vision with the substituting modalities touch [11]–[14] or audition [15]–[17].”
BackgroundColor improves “visual†acuity via sound
Levy-Tzedek, S., Riemer, D., & Amedi, A. (2014) · Frontiers in Neuroscience · doi:10.3389/fnins.2014.00358 · Cited by 30
“ORIGINAL RESEARCH ARTICLE published: 11 November 2014 doi: 10.3389/fnins.2014.00358 Color improves “visual” acuity via sound”
BackgroundMethodInfluentialSelf-citationImpact of a Vibrotactile Belt on Emotionally Challenging Everyday Situations of the Blind
Brandebusemeyer, C., Luther, A., König, S. U., König, P., & Kärcher, S. M. (2021) · Sensors · doi:10.3390/s21217384 · Cited by 33
Pissaloux, E. E., Velázquez, R., Hersh, M. A., & Uzan, G. (2016) · Journal of Navigation · doi:10.1017/s0373463316000461 · Cited by 32
“Information is provided to users in tactile and/or auditory modalities (e.g. CASBliP FP6 EU project (Dunai et al., 2014; Bujacz et al., 2012; Maidenbaum et al., 2014; Abboud et al., 2014)).”
BackgroundBayes-Like Integration of a New Sensory Skill with Vision
Negen, J., Wen, L., Thaler, L., & Nardini, M. (2018) · Scientific Reports · doi:10.1038/s41598-018-35046-7 · Cited by 34
“…(Kolarik et al., 2014; Stroffregen & Pittenger, 1995; Thaler & Goodale, 2016) and devices translating information from one modality into another (Abboud et al., 2014; Maidenbaum et al., 2014; Meijer, 1992) might not only hold promise for people with complete sensory loss (e.g. total blindness),…”
Eckert, M., Blex, M., & Friedrich, C. M. (2018) · International Joint Conference on Biomedical Engineering Systems and Technologies (BIOSTEC) · doi:10.5220/0006655605550561 · Cited by 33
“Red for Reggae Organ, Green for Rapmans Reed, Blue for Brass Instruments, White for Choir and Yellow for String Instruments (Abboud et al., 2014).”
BackgroundSoundSight: a mobile sensory substitution device that sonifies colour, distance, and temperature
Hamilton-Fletcher, G., Alvarez, J., Obrist, M., & Ward, J. (2022) · Journal on Multimodal User Interfaces · doi:10.1007/s12193-021-00376-w · Cited by 28
“Beyond the vOICe, this spectrogram-like approach of using pitch-height, panning-laterality, and loudness-brightness has underpinned many other SSD design choices with minor variations, such as presenting the information all-atonce (PSVA [19]; The Vibe [20]), musically (SmartSight [21]), or adding colour information th…”
BackgroundObject recognition for blind people based on features extraction
Jabnoun, H., Benzarti, F., & Amiri, H. (2014) · IEEE International Conference on Image Processing, Applications and Systems (IPAS) · doi:10.1109/ipas.2014.7043293 · Cited by 28
“In this section, we present some vision substitution modalities [10], [7] and their functionalities.”
BackgroundNau, A., Chan, K. C., & Murphy, M. (2015) · Neural Regeneration Research · doi:10.4103/1673-5374.169612 · Cited by 29
“The vOICe (Metamodal, Inc.), and the spinoff software program known as EyeMusic (Abboud et al., 2014) both translate visual stimuli into soundscapes.”
MethodMaidenbaum, S., Levy‐Tzedek, S., Chebat, D., Namer-Furstenberg, R., & Amedi, A. (2014) · Multisensory Research · doi:10.1163/22134808-00002463 · Cited by 29
Self-citationObject recognition for blind people using portable camera
Mohane, V., & Gode, C. S. (2016) · World Conference on Futuristic Trends in Research and Innovation for Social Welfare (Startup Conclave) · doi:10.1109/startup.2016.7583932 · Cited by 25
“Recently, several electronic assisting devices [2], [3], [4], [5], [6], [7], [8] & [9] developed for these persons but they cannot handle product recognition efficiently.”
BackgroundWaist-up protection for blind individuals using the EyeCane as a primary and secondary mobility aid
Buchs, G., Simon, N., Maidenbaum, S., & Amedi, A. (2017) · Restorative Neurology and Neuroscience · doi:10.3233/rnn-160686 · Cited by 35
Self-citationVisual substitution system for blind people based on SIFT description
Jabnoun, H., Benzarti, F., & Amiri, H. (2014) · International Conference of Soft Computing and Pattern Recognition (SoCPaR) · doi:10.1109/socpar.2014.7008023 · Cited by 24
“In this section, we present some vision substitution modalities [9], [7] and their functionalities”
BackgroundTal, Z., Geva, R., & Amedi, A. (2015) · NeuroImage · doi:10.1016/j.neuroimage.2015.11.058 · Cited by 28
“In a recent study by Abboud et al. (2015) blind subjects were presented with such Roman script shapes encoded to sounds by a novel visual-to-music sensory-substitution device developed in our lab (EyeMusic, Abboud et al., 2014).”
Self-citationVisual objects in the auditory system in sensory substitution: How much information do we need?
Brown, D. J., Simpson, A. J. R., & Proulx, M. J. (2014) · Multisensory Research · doi:10.1163/22134808-00002462 · Cited by 23
Maidenbaum, S., Arbel, R., Buchs, G., Shapira, S., & Amedi, A. (2014) · Mediterranean Conference on Control and Automation (MED) · doi:10.1109/med.2014.6961368 · Cited by 25
Self-citationKaur, B., & Bhattacharya, J. (2019) · Journal of Electronic Imaging · doi:10.1117/1.jei.28.1.013031 · Cited by 29
Heimler, B., Pavani, F., & Amedi, A. (2018) · Oxford University Press eBooks · doi:10.1093/oso/9780190880545.003.0015 · Cited by 22
Self-citationMulti-Sensory Color Code Based on Sound and Scent for Visual Art Appreciation
Quero, L. C., Lee, C.-H., & Cho, J. (2021) · Electronics · doi:10.3390/electronics10141696 · Cited by 24
“EyeMusic [14] conveys color information using a musical instrument’s timbre for each of the white (choir), blue (brass), red (Reggae organ), green (Rapman’s reed), and yellow (string) colors.”
BackgroundNeuroplasticity: Unexpected Consequences of Early Blindness
Murray, M. M., Matusz, P. J., & Amedi, A. (2015) · Current Biology · doi:10.1016/j.cub.2015.08.054 · Cited by 21
“‘soundscapes’ that individuals learn to use to see [12,14,15].”
Self-citationMusical Vision: an interactive bio-inspired sonification tool to convert images into music
Polo, A., & Sevillano, X. (2018) · Journal on Multimodal User Interfaces · doi:10.1007/s12193-018-0280-4 · Cited by 21
“More recently, EyeMusic [14,15] used a head-mounted camera to allow blindfolded sighted trained participants to grab physical objects usingmusic obtained from low resolution images, using instruments to codify up to four colors.”
MethodOn the Evaluation of Novel Sonification Techniques for Non-Visual Shape Exploration
Mascetti, S., Gerino, A., Bernareggi, C., & Picinali, L. (2017) · ACM Transactions on Accessible Computing · doi:10.1145/3046789 · Cited by 16
An investigation into the effectiveness of using acoustic touch to assist people who are blind
Zhu, H. Y., Hossain, S. N., Jin, C., Singh, A. K., Nguyen, M. T. D., Deverell, L., Nguyen, V., Gates, F. S., Fernandez, I. G., Melencio, M. V., et al. (2023) · PLoS ONE · doi:10.1371/journal.pone.0290431 · Cited by 16
Scene-Aware Background Music Synthesis
Wang, Y., Liang, W. G., Li, W., Li, D., & Yu, L.-F. (2020) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3394171.3413894 · Cited by 19
“, instrument [2], pitch and loudness [10].”
Maimon, A., Yizhar, O., Buchs, G., Heimler, B., & Amedi, A. (2022) · Neuropsychologia · doi:10.1016/j.neuropsychologia.2022.108305 · Cited by 20
“The soundscape is constructed by “sweeping” the image by column from left to right, with a novel sound frequency combination representing the image in the soundscape using a conversion algorithm specific to the SSD (Abboud et al., 2014).”
MethodSelf-citationA Sensory Substitution Device Inspired by the Human Visual System
Osinski, D., & Hjelme, D. R. (2018) · 2018 11th International Conference on Human System Interaction (HSI) · doi:10.1109/hsi.2018.8431078 · Cited by 15
Interoceptive training with real‐time haptic versus visual heartbeat feedback
Dobrushina, O. R., Tamim, Y., Wald, I. Y., Maimon, A., & Amedi, A. (2024) · Psychophysiology · doi:10.1111/psyp.14648 · Cited by 21
Self-citationRichardson, M., Esenkaya, T., Petrini, K., & Proulx, M. J. (2020) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376184 · Cited by 21
“This, in turn, can help further reveal the brain’s representation of, and interaction with, space [1, 3].”
BackgroundBrown, D. J., Simpson, A. J. R., & Proulx, M. J. (2015) · Frontiers in Psychology · doi:10.3389/fpsyg.2015.01522 · Cited by 18
Guidelines for quantitative and qualitative studies of sensory substitution experience
Kałwak, W., Reuter, M., Łukowska, M., Majchrowicz, B., & Wierzchoń, M. (2018) · Adaptive Behavior · doi:10.1177/1059712318771690 · Cited by 14
“, 2012) and congenital (Abboud et al., 2014) sensory deprivation improve SS adaptation and might result in better task performance.”
BackgroundMethodInfluentialVisual Echolocation Concept for the Colorophone Sensory Substitution Device Using Virtual Reality
Bizoń-Angov, P., Osiński, D., Wierzchoń, M., & Konieczny, J. (2021) · Sensors · doi:10.3390/s21010237 · Cited by 15
“[4] is based on a camera mounted on glasses and bone-conductive headphones.”
BackgroundInfluentialMultisensory perception and decision-making with a new sensory skill.
Negen, J., Bird, L., Slater, H., Thaler, L., & Nardini, M. (2023) · Journal of Experimental Psychology Human Perception & Performance · doi:10.1037/xhp0001114 · Cited by 17
Buchs, G., Heimler, B., & Amedi, A. (2019) · Multisensory Research · doi:10.1163/22134808-20181327 · Cited by 15
“The EyeMusic SSD The EyeMusic visual-to-auditory SSD transforms visual images into audio soundscapes (Abboud et al., 2014; Levy-Tzedek et al., 2012).”
BackgroundMethodInfluentialSelf-citationMultimodal Perception of Histological Images for Persons Who Are Blind or Visually Impaired
Zhang, T., Duerstock, B. S., & Wachs, J. P. (2017) · ACM Transactions on Accessible Computing · doi:10.1145/3026794 · Cited by 17
“Due to the limitation of differentiating among various musical instruments, the system only conveyed five colors: white, blue, red, green, and yellow [Abboud et al. 2014].”
BackgroundA self-training program for sensory substitution devices
Buchs, G., Haimler, B., Kerem, M., Maidenbaum, S., Braun, L., & Amedi, A. (2021) · PLoS ONE · doi:10.1371/journal.pone.0250281 · Cited by 16
“In the present study we used the EyeMusic a visual-to-auditory SSD developed in our lab, which transforms whole-visual images into auditory inputs, termed soundscapes, preserving shape, location and even color of the objects in the scene [3].”
BackgroundMethodInfluentialSelf-citationSmart Cane Developed with DFSS, QFD, and SDE for the Visually Impaired
Frizziero, L., Liverani, A., Donnici, G., Papaleo, P., & Leon-Cardenas, C. (2021) · Inventions · doi:10.3390/inventions6030058 · Cited by 15
“[29]; a vibrotactile device that can translate facial expressions by Buimer et al.”
BackgroundMultisensory and spatial processes in sensory substitution
Auvray, M. (2019) · Restorative Neurology and Neuroscience · doi:10.3233/rnn-190950 · Cited by 17
“The four main visual-to-auditory devices (e.g., the vOICe, Meijer, 1992; The PSVA, Capelle et al., 1998; Vibe, Hanneton, Auvray, & Durette, 2010; the EyeMusic, Abboud et al., 2014) have been designed by using the correspondence between direction of pitch and direction of movement (higher-higher; lower-lower).”
Deep Learning-Based Optimization of Visual–Auditory Sensory Substitution
Kim, M., Park, Y., Moon, K., & Jeong, C. Y. (2023) · IEEE Access · doi:10.1109/access.2023.3243641 · Cited by 12
“EyeMusic [17] allows the user to distinguish between five colors, whereas See ColOr [18] delivers seven colors.”
BackgroundMethodNetzer, O., Heimler, B., Shur, A., Behor, T., & Amedi, A. (2021) · Scientific Reports · doi:10.1038/s41598-021-88595-9 · Cited by 12
Self-citationArbel, R., Heimler, B., & Amedi, A. (2022) · Scientific Reports · doi:10.1038/s41598-022-08187-z · Cited by 14
Self-citationBordeau, C., Scalvini, F., Migniot, C., Dubois, J., & Ambard, M. (2023) · Frontiers in Psychology · doi:10.3389/fpsyg.2023.1079998 · Cited by 14
“The EyeMusic SSD (Abboud et al., 2014) uses a pentatonic musical scale with 24 frequencies from 65.”
MethodInfluentialVisuo-spatial imagery in dreams of congenitally and early blind: a systematic review
Ilić, K., Bertani, R., Lapteva, N., Drakatos, P., Delogu, A., Raheel, K., Soteriou, M., Mutti, C., Steier, J. S., Carmichael, D. W., et al. (2023) · Frontiers in Integrative Neuroscience · doi:10.3389/fnint.2023.1204129 · Cited by 13
“(2019) employed EyeMusic, an sensory substitution devices which uses different musical instruments to convey color information (Abboud et al., 2014).”
Background'Visual’ parsing can be taught quickly without visual experience during critical periods
Reich, L., & Amedi, A. (2015) · Scientific Reports · doi:10.1038/srep15359 · Cited by 12
Self-citationDevelopment of a versatile assistive system for the visually impaired based on sensor fusion
Botezatu, N. A., Caraiman, S., Rzeszotarski, D., & Strumiłło, P. (2017) · International Conference on System Theory, Control and Computing (ICSTCC) · doi:10.1109/icstcc.2017.8107091 · Cited by 13
“Sensory substitution devices are designed to convey visual information to the visually impaired by substituting visual information into one of their intact senses [3-9].”
BackgroundVision sensory substitution to aid the blind in reading and object recognition
Bhat, P. G., Rout, D. K., Subudhi, B. N., & Veerakumar, T. (2017) · International Conference on Image Information Processing (ICIIP) · doi:10.1109/iciip.2017.8313754 · Cited by 12
“Some existing technologies like EyeMusic [9], visotoner [10], Google glass and Zungle glasses also show some potential but cannot be improved to specifically suit these needs of the blind and visually impaired.”
BackgroundOsiński, D., Łukowska, M., Hjelme, D. R., & Wierzchoń, M. (2021) · Sensors · doi:10.3390/s21217351 · Cited by 11
“The existing color sonification methods applied in SSDs can be divided into three categories: the first category contains systems that directly associate light frequency with sound frequency [21,22], the second category systems use associations between a predefined color category and the presented sound [20,46] (e.”
MethodMental Imagery in Dreams of Congenitally Blind People
Kang, J., Bertani, R., Raheel, K., Soteriou, M., Rosenzweig, J., Valentín, A., Goadsby, P. J., Tahmasian, M., Moran, R., Ilić, K., et al. (2023) · Brain Sciences · doi:10.3390/brainsci13101394 · Cited by 12
The mechanism of human color vision and potential implanted devices for artificial color vision
Zhang, B., Zhang, R., Zhao, J., Yang, J., & Xu, S. (2024) · Frontiers in Neuroscience · doi:10.3389/fnins.2024.1408087 · Cited by 16
Gui, W., Li, B., Yuan, S., Rizzo, J., Sharma, L., Feng, C., Tzes, A. P., & Fang, Y. (2019) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros40897.2019.8967647 · Cited by 12
Proulx, M. J., Brown, D. J., Esenkaya, T., Leveson, J. B., Todorov, O. S., Watson, S. H., & de Sousa, A. A. (2020) · Applied Ergonomics · doi:10.1016/j.apergo.2020.103072 · Cited by 12
Spatial Haptics: A Sensory Substitution Method for Distal Object Detection Using Tactile Cues
Wald, I. Y., Degraen, D., Maimon, A., Keppel, J., Schneegaß, S., & Malaka, R. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3714083 · Cited by 15
“Following these known fundamentals of sensory substitution and perceptual learning, it is likely that users’ performance using our system could implicitly improve with extended use, or be trained for better accuracy [1, 5, 6, 8, 12, 14, 26, 59– 61, 64, 70].”
Hofstetter, S., Zuiderbaan, W., Heimler, B., Dumoulin, S. O., & Amedi, A. (2021) · NeuroImage · doi:10.1016/j.neuroimage.2021.118029 · Cited by 10
“MaMe engaged during scanning, he was asked to respond when a bar appeared in red rather than white (colors are represented by a different type of sound (Abboud et al., 2014)).”
InfluentialSelf-citationEvaluating the effectiveness of sonification in science education using Edukoi
Fovino, L. G. N., Zanella, A., Mascolo, L. D., Ginolfi, M., Carpita, N., Manuncola, F. T., & Grassi, M. (2024) · Personal and Ubiquitous Computing · doi:10.1007/s00779-024-01809-5 · Cited by 16
“In the literature, we find several studies on the topic of shape sonification with various sound mappings [42, 47–49].”
Use of a sonification system for science learning by people who are blind
Lahav, O., Chagab, N., & Talis, V. (2016) · Journal of Assistive Technologies · doi:10.1108/jat-11-2015-0032 · Cited by 11
Arbel, R., Heimler, B., & Amedi, A. (2022) · Frontiers in Neuroscience · doi:10.3389/fnins.2022.921321 · Cited by 9
“These SSDs transform visual information into audition using a specific algorithm that maintains core visual features such as objects’ shapes, locations, and even color (Meijer, 1992; Abboud et al., 2014).”
MethodSelf-citationMaimon, A., Wald, I. Y., Oz, M. B., Codron, S., Netzer, O., Heimler, B., & Amedi, A. (2023) · Frontiers in Human Neuroscience · doi:10.3389/fnhum.2022.1058093 · Cited by 11
“New technologies incorporate an element of color into echolocation-inspired devices, such as the EyeMusic (Abboud et al., 2014) and the Colorphone (Bizoń-Angov et al.”
BackgroundMethodInfluentialSelf-citationMobile Video-to-Audio Transducer and Motion Detection for Sensory Substitution
Ambard, M., Benezeth, Y., & Pfister, P. (2015) · Frontiers in ICT · doi:10.3389/fict.2015.00020 · Cited by 13
“EyeMusic (Levy-Tzedek et al., 2012, 2014; Abboud et al., 2014; Maidenbaum et al., 2014a) is based on periodic left-to-right horizontal scans of images in which the sounds of musical instruments are produced depending on the colors of the pixels.”
BackgroundAnalysis and Validation of Cross-Modal Generative Adversarial Network for Sensory Substitution
Kim, M., Park, Y., Moon, K., & Jeong, C. Y. (2021) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph18126216 · Cited by 8
“[10] developed the EyeMusic system, which provided a musical image representation preserving the shape and integrating color information.”
BackgroundBuchs, G., Maidenbaum, S., Levy-Tzedek, S., & Amedi, A. (2015) · Restorative Neurology and Neuroscience · doi:10.3233/rnn-150592 · Cited by 8
“The EyeMusic has been used before in several other projects, concerning the identification of shapes and objects (Abboud et al., 2014; Maidenbaum, Arbel, Buchs, Shapira, & Amedi, 2014) exploration of the meta-modal basis of number representation in the brain (Abboud, Maidenbaum, Dehaene, & Amedi, 2015), and swift cros…”
BackgroundMethodSelf-citationJiang, P., Kent, C. D., & Rossiter, J. M. (2024) · PLoS ONE · doi:10.1371/journal.pone.0299213 · Cited by 10
“For example, some devices use musical notes to communicate color and shape information [13].”
BackgroundHelping Blind People Grasp: Evaluating a Tactile Bracelet for Remotely Guiding Grasping Movements
Powell, P., Pätzold, F., Rouygari, M., Furtak, M., Kärcher, S. M., & König, P. (2024) · Sensors · doi:10.3390/s24092949 · Cited by 10
“Several devices have achieved success in allowing their users to ”see” their surroundings by substituting feedback from other senses for the missing visual channel (e.g., [17–20]).”
BackgroundMaimon, A., Wald, I. Y., Snir, A., Oz, M. B., & Amedi, A. (2024) · PLoS ONE · doi:10.1371/journal.pone.0310033 · Cited by 10
Self-citationThe MAPS: Toward a Novel Mobility Assistance System for Visually Impaired People
Romeo, K., Pissaloux, E. E., Gay, S. L., Truong, N.-T., & Djoussouf, L. (2022) · Sensors · doi:10.3390/s22093316 · Cited by 11
“The vOICe and the EyeMusic offer a higher resolution, but they comprise complex coding that makes them more difficult to use and they, consequently, require many hours of training.”
InfluentialNewly learned novel cues to location are combined with familiar cues but not always with each other.
Aston, S., Beierholm, U., & Nardini, M. (2022) · Journal of Experimental Psychology Human Perception & Performance · doi:10.1037/xhp0001014 · Cited by 8
Łukowska, M., Kałwak, W., Osiński, D., Janik, J., & Wierzchoń, M. (2022) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2022.102925 · Cited by 11
Shape detection beyond the visual field using a visual-to-auditory sensory augmentation device
Shvadron, S., Snir, A., Maimon, A., Yizhar, O., Harel, S., Poradosu, K., & Amedi, A. (2023) · Frontiers in Human Neuroscience · doi:10.3389/fnhum.2023.1058617 · Cited by 8
“Visual-to-auditory transformation of the current study, Vision360, is based on techniques of the EyeMusic technology (Abboud et al., 2014), in which pixels along the Y-axis (bottom to top) are converted to consecutive pitches along a pentatonic scale.”
MethodSelf-citationMaidenbaum, S., Abboud, S., Buchs, G., & Amedi, A. (2015) · IEEE Conference on Virtual Reality and 3D User Interfaces (VR) · doi:10.1109/vr.2015.7223381 · Cited by 8
“The EyeMusic is a visual-to-auditory SSD developed by the Amedi team, described in detail in [5], and used for various tasks in real-world and experimental.”
BackgroundMethodSelf-citation
Metadata
- Authors
- Abboud, S., Hanassy, S., Levy-Tzedek, S., Maidenbaum, S., & Amedi, A.
- Affiliations
- Hebrew University of Jerusalem (IL)
- Department of Medical Neurobiology, The Institute for Medical Research Israel-Canada, Faculty of Medicine, The Hebrew University of Jerusalem, Jerusalem, Israel
- Corresponding author
- Amedi, Amir
- Venue
- Restorative Neurology and Neuroscience
- Publisher
- SAGE Publications
- Published
- 2014-01-01
- Volume
- 32
- Issue
- 2
- Pages
- 247-257
- ISSN
- 0922-6028 (ISSN-L)
- 1878-3627
- OpenAlex
- W1922152898
- Cited by
- 199
- References
- 43
- Citations by year
- 2026: 11 · 2025: 14 · 2024: 16 · 2023: 17 · 2022: 20 · 2021: 16 · 2020: 18 · 2019: 17 · 2018: 10 · 2017: 12 · 2016: 17 · 2015: 22 · 2014: 9
- FWCI
- 11.15
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Restorative Neurology and Neuroscience (publishedVersion, cc-by-nc)
- PubMed (publishedVersion)
- Funders
- James S. McDonnell Foundation (220020284)
- Israel Science Foundation (ISF 1684/08, 1684/08)
- FP7 Ideas: European Research Council (310809)
- Topics
- Life Sciences › Neuroscience › Cognitive Neuroscience › Tactile and Sensory Interactions
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Multisensory perception and integration
- Health Sciences › Health Professions › Occupational Therapy › Assistive Technology in Communication and Mobility
- Keywords
- sensory substitution devices, color perception, shape perception, blind users, pentatonic scale, visual rehabilitation
- MeSH terms
- Acoustic Stimulation, Adult, Auditory Perception, Blindness, Female, Humans, Male, Middle Aged, Neuronal Plasticity, Sensation, Persons with Visual Disabilities, Education of Persons with Visual Disabilities
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.