EyeMusic: Introducing a “visual” colorful experience for the blind using auditory sensory substitution
Open accessThe DOI matches its Crossref record; 42 of the 43 references it cites were found in a registry or an index.
Abstract
Purpose: Sensory-substitution devices (SSDs) provide auditory or tactile representations of visual information. These devices often generate unpleasant sensations and mostly lack color information. We present here a novel SSD aimed at addressing these issues. Methods: We developed the EyeMusic, a novel visual-to-auditory SSD for the blind, providing both shape and color information. Our design uses musical notes on a pentatonic scale generated by natural instruments to convey the visual information in a pleasant manner. A short behavioral protocol was utilized to train the blind to extract shape and color information, and test their acquired abilities. Finally, we conducted a survey and a comparison task to assess the pleasantness of the generated auditory stimuli. Results: We show that basic shape and color information can be decoded from the generated auditory stimuli. High performance levels were achieved by all participants following as little as 2–3 hours of training. Furthermore, we show that users indeed found the stimuli pleasant and potentially tolerable for prolonged use. Conclusions: The novel EyeMusic algorithm provides an intuitive and relatively pleasant way for the blind to extract shape and color information. We suggest that this might help facilitating visual rehabilitation because of the added functionality and enhanced pleasantness.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Using techniques such as computer-synthesized speech (e.g., object name and direction), spatial audio, head rotation, and modulated audio (pitch and amplitude modulation), past works have found success in using auditory sensory augmentation to provide users with accurate spatial information of their surrounding environment with very little training [15, 19–22].”
From An investigation into the effectiveness of using acoustic touch to assist people who are blind (2023) · cites this paper as [20]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“The participants' feedback or task performance is used as an indicator to assess the efficiency of the conversion method in the conventional assessment [17], [40].”
From Deep Learning-Based Optimization of Visual–Auditory Sensory Substitution (2023) · cites this paper as [17]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Quattoni, A., & Torralba, A. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206537 · Cited by 1,463
Vision substitution by tactile image projection
BACH-Y-RITA, P., COLLINS, C. C., SAUNDERS, F. A., WHITE, B., & SCADDEN, L. (1969) · Nature · doi:10.1038/221963a0 · Cited by 956 · cited 2×
“stimulators which vibrate against the skin of the back conveying shapes as captured from a television camera (Bach-y-Rita et al., 1969); and the “Tongue Display Unit” (TDU), generating electro-tactile pulses displayed on the tongue (Kaczmarek, 2011).”
An experimental system for auditory image representations
Meijer, P. B. L. (1992) · IEEE Transactions on Biomedical Engineering · doi:10.1109/10.121642 · Cited by 900 · cited 5×
“In the visual-to-auditory domain, “The vOICe” introduced the sweep-line technique that generated auditory representations, called soundscapes, which map the xaxis of captured images to the time domain and the y-axis to the frequency domain (Meijer, 1992).”
InfluentialDecomposing a scene into geometric and semantically consistent regions
Gould, S., Fulton, R., & Koller, D. (2009) · 2009 IEEE 12th International Conference on Computer Vision · doi:10.1109/iccv.2009.5459211 · Cited by 672 · cited 1×
“We designed an online task with 30 stimuli consisting of 3 categories: indoor scenes (Adapted from Quattoni et al., 2009), outdoor scenes (Adapted from Gould et al., 2009) and simple shapes that we designed.”
Tactile Acuity is Enhanced in Blindness
Goldreich, D., & Kanics, I. M. (2003) · The Journal of Neuroscience · doi:10.1523/jneurosci.23-08-03439.2003 · Cited by 427 · cited 1×
“These devices rely on the improved auditory and tactile sensations of the blind (For examples see, Goldreich et al., 2003; Gougoux et al., 2004; Sathian et al., 2010) and aim to generate a shape-preserving representation of the environment as captured by an input device, e.g., a camera.”
Pitch discrimination in the early blind
Gougoux, F., Lepore, F., Lassonde, M., Voss, P., Zatorre, R. J., & Belin, P. (2004) · Nature · doi:10.1038/430309a · Cited by 422
Striem-Amit, E., Cohen, L., Dehaene, S., & Amedi, A. (2012) · Neuron · doi:10.1016/j.neuron.2012.08.026 · Cited by 294 · cited 3×
“In contrast, tasks that require higher resolution such as assessing facial expressions (Striem-Amit et al., 2012b) could benefit from an SSD with a higher resolution, such as the vOICe, which will have the upper hand.”
Self-citationAuvray, M., Hanneton, S., & O'Regan, J. K. (2007) · Perception · doi:10.1068/p5631 · Cited by 268 · cited 1×
“Various psychophysical experiments have demonstrated the ability of blind users to perform object and letter identification and discrimination (Auvray et al., 2007; Brown et al., 2011; Striem-Amit et al., 2012a), recognize patterns (Poirier et al., 2007), pair socks by color (Bologna et al., 2009),…”
Visual experiences in the blind induced by an auditory sensory substitution device
Ward, J., & Meijer, P. (2010) · Consciousness and Cognition · doi:10.1016/j.concog.2009.10.006 · Cited by 247
Quattoni, A., & Torralba, A. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvprw.2009.5206537 · Cited by 228
Cited by
Items 101–198 of 198
Maidenbaum, S., & Amedi, A. (2015) · doi:10.1109/vaat.2015.7155404 · Cited by 9
“For a more detailed explanations about the transformation see [19].”
Self-citationAutoencoding sensory substitution
Tóth, V., & Parkkonen, L. (2019) · Unpublished · doi:10.13140/rg.2.2.10576.87048 · Cited by 7
“EyeMusic [151] combines See ColOr and TV, performing color clustering, synthesizing soundscapes column-by-column, and ceiling the frequency at 1568 Hz, assuring the pleasantness of the auditory stimulus; albeit, the frequency cut-off reduces the auditory coding space by half.”
BackgroundMethodRayes, R., Mazorow, R. N., Mrotek, L. A., & Scheidt, R. A. (2023) · Sensors · doi:10.3390/s23125455 · Cited by 7
Ruan, L., Hamilton-Fletcher, G., Beheshti, M., Hudson, T. E., Porfiri, M., & Rizzo, J. (2025) · Disability and Rehabilitation Assistive Technology · doi:10.1080/17483107.2025.2463541 · Cited by 10
Sensory augmentation to aid training with retinal prostheses
Kvansakul, J. G. S., Hamilton, L. H. W., Ayton, L. N., McCarthy, C. D., & Petoe, M. A. (2020) · Journal of Neural Engineering · doi:10.1088/1741-2552/ab9e1d · Cited by 7
Baker, R. M., Ramos, K., & Turner, J. R. (2018) · Irish Journal of Technology Enhanced Learning · doi:10.22554/ijtel.v4i1.51 · Cited by 7
Arbel, R., Heimler, B., & Amedi, A. (2020) · PLoS ONE · doi:10.1371/journal.pone.0242619 · Cited by 6
“introduction of color and shapes through the teaching of reading a script can potentially be more engaging and enjoyable for users than the use of simple lines and geometrical shapes in the initial steps of SSD training as typically carried out [44].”
BackgroundSelf-citationHeimler, B., & Amedi, A. (2019) · Elsevier eBooks · doi:10.1016/b978-0-12-812492-5.00015-2 · Cited by 9
Self-citationDesign and Implementation of a Real-Time Color Recognition System for the Visually Impaired
Samara, M., Deriche, M. A., Al-Sadah, J., & Osais, Y. E. (2022) · Arabian Journal for Science and Engineering · doi:10.1007/s13369-022-07506-w · Cited by 8
“Abboud et al. [35] developed The EyeMusic system, a novel visual to auditory sensory substitution system for color and shape information in an image.”
BackgroundResultLocalizing 3D motion through the fingertips: Following in the footsteps of elephants
Snir, A., Cieśla, K., Ozdemir, G., Vekslar, R., & Amedi, A. (2024) · iScience · doi:10.1016/j.isci.2024.109820 · Cited by 6
Self-citationSocialCueSwitch: Towards Customizable Accessibility by Representing Social Cues in Multiple Senses
Segal, J. I., Rodriguez, S., Raghavan, A., Baez, H. B., Jung, C., Collins, J., Azenkot, S., & Won, A. S. (2024) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3613905.3651109 · Cited by 11
“In social VR, interactions often rely on a combination of visual cues such as gestures, avatars, and text chats, and many tool sets focus on a primary sensory impairment such as displaying visual stimuli auditorily [1, 3, 11, 21].”
BackgroundExploring Eye-Tracking-Driven Sonification for the Visually Impaired
Dietz, M., Elgarf, M., Damian, I., & André, E. (2016) · Augmented Human International Conference · doi:10.1145/2875194.2875208 · Cited by 6
Towards a Natural User Interface to Support People with Visual Impairments in Detecting Colors
Mascetti, S., Rossetti, C., Gerino, A., Bernareggi, C., Picinali, L., & Rizzi, A. (2016) · Lecture notes in computer science · doi:10.1007/978-3-319-41264-1_23 · Cited by 6
“[12] propose EyeMusic, an application that enables the sonification of images captured through a camera.”
BackgroundVincent, M., Tang, H., Khoo, W. L., Zhu, Z., & Ro, T. (2016) · Multisensory Research · doi:10.1163/22134808-00002542 · Cited by 7
“…visual-to-auditory sensory substitution device called EyeMusic, which conveys visual information using musical notes on a pentatonic scale generated using a sweep-line technique where each image is processed column-by-column from left to right constructing a ‘soundscape’ (Abboud et al., 2014).”
MethodSnir, A., Cieśla, K., Vekslar, R., & Amedi, A. (2024) · iScience · doi:10.1016/j.isci.2024.110808 · Cited by 6
“…audio-tactile learning is due to the numerous similarities between these two senses, which is also the reason why the outcomes are more immediate, as compared to other multisensory solutions (see introduction; cf. long training regimes for audiovisual and visuo-tactile set-ups 39,53,54,56 ).”
Self-citationPerception of sonified representations of complex systems by people who are blind
Lahav, O., Kittany, J., Levy, S. T., & Furst, M. (2019) · Assistive Technology · doi:10.1080/10400435.2019.1666930 · Cited by 5
AI-Vision Towards an Improved Social Inclusion
Alashkar, R., ElSabbahy, M., Sabha, A., Abdelghany, M., Tlili, B., & Mounsef, J. (2020) · IEEE/ITU International Conference on Artificial Intelligence for Good (AI4G) · doi:10.1109/ai4g50087.2020.9311049 · Cited by 5
“Finally, Abboud et al. [18] develop a novel visual to audio SSD named EyeMusic.”
BackgroundRudenko, S., & Haahr, M. (2022) · AHFE international · doi:10.54941/ahfe1001639 · Cited by 6
“) Amedi’s App EyeMusic introduces a “visual” colorful experience for the blind using auditory sensory substitution, where “musical notes on a pentatonic scale generated by natural instruments (to) convey the visual information in a pleasant manner” (Abboud et al., 2014).”
BackgroundAnalysis of Multimodal Teaching of College English under the Background of Artificial Intelligence
Niu, L. (2022) · Security and Communication Networks · doi:10.1155/2022/3833106 · Cited by 5
RetractedStandardizing Visual Rehabilitation using Simple Virtual Tests
Maidenbaum, S., & Amedi, A. (2019) · International Conference on Virtual Rehabilitation · doi:10.1109/icvr46560.2019.8994431 · Cited by 6
“This device has already been used for a series of experiments including object and shape recognition, target reaching and exploring the neural correlates of number representation, for tasks in virtual environments and for real-world tasks such as finding a specific item in a noisy supermarket [19,26,32].”
MethodSelf-citationArbel, R., Heimler, B., & Amedi, A. (2023) · Neuropsychologia · doi:10.1016/j.neuropsychologia.2023.108685 · Cited by 5
Self-citationThermalSense: A Sensory Substitution System for Enabling Perception of Thermal Information
Wald, I. Y., Bronstein, Y., Maimon, A., & Amedi, A. (2025) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3733048 · Cited by 4
“The algorithm employs tones in the pentatonic scale, shown to be pleasant and harmonious even during the presentation of numerous simultaneous sounds [1].”
InfluentialSelf-citationCaesynth: Real-Time Timbre Interpolation and Pitch Control with Conditional Autoencoders
Puche, A. V., & Lee, S. (2021) · Machine learning for signal processing ... · doi:10.1109/mlsp52302.2021.9596414 · Cited by 6
Modern approaches to orientation and mobility
Wall, K. (2019) · Routledge eBooks · doi:10.4324/9781315111353-22 · Cited by 4
Sensory augmentation for a rapid motor task in a multisensory environment
Negen, J., Slater, H., & Nardini, M. (2023) · Restorative Neurology and Neuroscience · doi:10.3233/rnn-221279 · Cited by 4
“SSASys have already been shown repeatedly to help people make untimed judgements in unisensory tasks (e.g., Abboud et al., 2014; Auvray et al., 2007), such as using echolocation to sense distance (Thaler & Goodale, 2016).”
BackgroundDepth-To-Audio Sensory Substitution for Increasing the Accessibility of Virtual Environments
Maidenbaum, S., Chebat, D., Levy‐Tzedek, S., & Amedi, A. (2014) · Lecture notes in computer science · doi:10.1007/978-3-319-07437-5_38 · Cited by 4
“EyeMusic: Introducing a “visual” colorful experience for the blind using auditory sensory substitu- tion.”
BackgroundSelf-citationA wearable device for blind people to restore color perception
Cavadini, R., Cerina, L., Bracco, M. I., & Santambrogio, M. D. (2017) · IEEE International Forum on Research and Technology for Society and Industry (RTSI) · doi:10.1109/rtsi.2017.8065974 · Cited by 5
“However, as reported by [5], Sensory Substitution Devices (SSDs) are not widely used by the blind community, because they are considered expensive, bulky and unpleasant to be used and worn.”
BackgroundInvestigating techniques for gaining depth perception using visual-to-auditory sensory substitution*
de Klerk, J. C., Vogts, D., & Wesson, J. L. (2020) · South African Institute of Computer Scientists and Information Technologists · doi:10.1145/3410886.3410916 · Cited by 5
“This includes SSDs such as the vOICe [5], [6], the PSVA [7], EyeMusic [8] and MeloSee [9], [10].”
MethodPerception of color in primates: A conceptual color neurons hypothesis
Aseyev, N. (2023) · Biosystems · doi:10.1016/j.biosystems.2023.104867 · Cited by 4
Kruger, R., de Wet, F., & Niesler, T. (2023) · ACM Transactions on Accessible Computing · doi:10.1145/3584365 · Cited by 6
Ito, Y., Terada, T., & Tsukamoto, M. (2015) · International Conference on Information Integration and Web-based Applications & Services (iiWAS) · doi:10.1145/2837185.2837235 · Cited by 4
“This method includes Abboud’s method [5] mentioned in section 2 and the method that allocates the sounds to each coordinate of an image of a character.”
BackgroundMethodInfluentialPort, A., Kim, C., & Patel, M. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2005.13291 · Cited by 6
“Several other approaches to visual-to-audio sensory substitution have been studied which do not make use of machine learning.[1, 2, 6, 9, 3] Unfortunately, to our knowledge none of these models are available to the public except for the vOICe.”
BackgroundReal, S., & Araújo, Á. (2023) · Sensors · doi:10.3390/s23063219 · Cited by 5
“This constitutes one of the main families of SD, which includes we l-known visual-to-auditory SD projects such as vOICe, EyeMusic [23], or VAS, as we l as visual-to-tactile SD, e.g., BrainPort [24], etc.”
BackgroundInfluentialButorovа, A. S., & Sergeev, A. (2025) · The European Physical Journal Special Topics · doi:10.1140/epjs/s11734-025-01596-x · Cited by 3
“When a similar sonification algorithm is supplemented with color information, like in EyeMusic [48], the sensory–cognitive load of recognizing a given pattern increases.”
InfluentialDesign and Implementation of a Stick for Blind Humans Using Artificial Intelligence System and FPGA
Jawad, R., & Jawad, R. (2024) · International Conference on Electrical, Computer and Energy Technologies (ICECET) · doi:10.1109/icecet61485.2024.10698153 · Cited by 3
Bordeau, C., Scalvini, F., Migniot, C., Dubois, J., & Ambard, M. (2025) · Attention Perception & Psychophysics · doi:10.3758/s13414-025-03065-y · Cited by 3
“The work of Buchs et al. (2019) investigated the effect of irrelevant background sounds on the ability to perform a task with the EyeMusic SSD (Abboud et al., 2014).”
Reaching and Grasping With a Sensory Substitution Glove: Variability of Practice Effects
Bilal, S., Bongers, R. M., & Jacobs, D. M. (2025) · Journal of Motor Learning and Development · doi:10.1123/jmld.2024-0060 · Cited by 3
Software Design for Low-Latency Visuo-Auditory Sensory Substitution on Mobile Devices
Ambard, M. (2017) · Computer and Information Science · doi:10.5539/cis.v10n2p1 · Cited by 5
“EyeMusic (Levy-Tzedek et al., 2012; Abboud et al., 2014; Maidenbaum et al., 2014A; Levy-Tzedek et al., 2014) is an initiative which is also based on periodic left-to-right horizontal scans of images.”
BackgroundVBTones a visual to auditory device for the blind
Riahi, N., Mirhosseini, S. F., & Mehrbani, E. (2017) · International Journal of Advanced Computer Research · doi:10.19101/ijacr.2017.733033 · Cited by 5
“This paper has presented a research on visual to auditory conversion algorithms which in addition to specific applications in rehabilitation of the blind, can be used as a tool for studying the cross modal performance of the brain in addition to the investigation the navigating capabilities based on the virtual enviro…”
BackgroundMethodInfluentialA systematic computerized training program for using Sensory Substitution Devices in real-life
Netzer, O., Buchs, G., Heimler, B., & Amedi, A. (2019) · International Conference on Virtual Rehabilitation · doi:10.1109/icvr46560.2019.8994518 · Cited by 3
“During the object-recognition task, participants heard EyeMusic stimuli belonging to the given category, and they had to assign the object to one of the 3 defined sub-divisions by pressing the corresponding button on the computer.”
BackgroundInfluentialSelf-citationAudio-Visual Thumble (AVT): A low-vision rehabilitation device using multisensory feedbacks
Ahmad, H., Tonelli, A., Crepaldi, M., Martolini, C., Capris, E., & Gori, M. (2020) · Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC) · doi:10.1109/embc44109.2020.9175475 · Cited by 4
Applying a novel visual-to-touch sensory substitution for studying tactile reference frames
Yizhar, O., Buchs, G., Heimler, B., Friedman, D. A., & Amedi, A. (2021) · Scientific Reports · doi:10.1038/s41598-021-90132-7 · Cited by 3
Self-citationMaidenbaum, S. (2015) · International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS) · doi:10.1145/2700648.2811324 · Cited by 3
Self-citationVirtually zooming-in with sensory substitution for blind users
Buchs, G., Maidenbaum, S., Amedi, A., & Levy‐Tzedek, S. (2015) · International Conference on Virtual Rehabilitation · doi:10.1109/icvr.2015.7358613 · Cited by 3
“The translation of visual input into audio stimuli was accomplished with the EyeMusic algorithm [8].”
BackgroundMethodSelf-citationSonorous cartography for sighted and blind people
Josselin, D., Roussel, D., Boularouk, S., Saidi, A., Matrouf, D., Bonin, O., & Altman, E. (2016) · HAL · Cited by 3
Ellenberger, C. (2016) · Handbook of clinical neurology · doi:10.1016/b978-0-444-53486-6.00045-4 · Cited by 3
A color recognition system for the visually impaired people
Samara, M., Al-Sadah, J., Driche, M., & Osais, Y. E. (2017) · IEEE International Conference on Engineering Technologies and Applied Sciences (ICETAS) · doi:10.1109/icetas.2017.8277848 · Cited by 4
“[8] developed a novel visual to audio sensory substitution device called the EyeMusic.”
BackgroundPrototyping and Evaluating Sensory Substitution Devices by Spatial Immersion in Virtual Environments
Guezou‐Philippe, A., Huet, S., Pellerin, D., & Graff, C. (2018) · International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications (VISIGRAPP) · doi:10.5220/0006637705960602 · Cited by 4
“These avatars are steered using a video game controller with a tactile feedback (Khoo et al., 2012), the Phantom device with haptic and sonic feedbacks (Lahav et al., 2015), or keyboard keys (Maidenbaum et al., 2016) with sonic feedback generated with the EyeMusic SSD (Abboud et al., 2014).”
MethodRassia, K. E. K., Liapi, L., & Pezaris, J. S. (2026) · Progress in Retinal and Eye Research · doi:10.1016/j.preteyeres.2026.101494 · Cited by 2
A Novel mapping for visual to auditory sensory substitution
Mirhosseini, S. F., Mehrbani, E., & Riahi, N. (2020) · Iranian Conference on Electrical Engineering · doi:10.1109/icee50131.2020.9260805 · Cited by 2
Butorovа, A. S., Koryukin, E. A., Khomenko, N. M., & Sergeev, A. (2024) · Современные технологии в медицине · doi:10.17691/stm2024.16.4.03 · Cited by 3
Moon, K.-D., Park, Y. K., Kim, M., & Jeong, C. Y. (2025) · IEEE Transactions on Neural Systems and Rehabilitation Engineering · doi:10.1109/tnsre.2025.3548942 · Cited by 2
“Sensory Substitution (SS) technologies, such as Visual-to-Auditory SS (V2A-SS) [2], [3], [4], [5] or Visual-to-Tactile SS (V2T-SS) [6], [7], have been investigated to restore the function of damaged SOs by transmitting information to the brain through a substitute SO instead of the damaged SO,…”
Applying cross-modal plasticity principles in auditory training applications
Huang, Q., Stawarz, K., Zhao, L., Yang, S., Xie, W., Song, F., & Liu, H. (2025) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2025.103570 · Cited by 2
Matsuda, S., Nakagawa, Y., Funazaki, Y., Matsuyama, N., Nakamura, S., Komatsu, T., Torii, T., Sumikawa, R., & Takao, H. (2022) · Procedia Computer Science · doi:10.1016/j.procs.2022.09.280 · Cited by 2
Harris, J. M., Hughes, A. E., Occelli, V., & Strong, S. (2024) · Visual Cognition · doi:10.1080/13506285.2023.2251182 · Cited by 3
Participatory Design Using Sensory Substitution Devices with Tactile and Audio Feedback
Tuson, E., Hughson, S., Zymaris, C., & King, R. (2017) · International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS) · doi:10.1145/3132525.3134826 · Cited by 2
“Likewise, TC has had little success with auditory simulation systems, ruling out the use of a system similar to EyeMusic [1].”
BackgroundA Low Resolution Haptic Interface for Interactive Applications
Fakhri, B., Sharma, S., Soni, B., Chowdhury, A., McDaniel, T. L., & Panchanathan, S. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-23563-5_19 · Cited by 2
Haptics for Sensory Substitution
Fakhri, B., & Panchanathan, S. (2019) · Springer eBooks · doi:10.1007/978-3-030-34230-2_4 · Cited by 3
Colorophone 2.2 – A Spatial, Real-Time Color-to-Sound Sensory Substitution DevicePreprint
Osiński, D., Łukowska, M., Kałwak, W., Wierzchoń, M., & Hjelme, D. R. (2024) · doi:10.36227/techrxiv.21385140.v2 · Cited by 2
Graphic-to-Sound Sonification for Visual and Auditory Communication Design
Joo, W. (2022) · Audio Mostly Conference (AM) · doi:10.1145/3561212.3561214 · Cited by 1
COLOR PERCEPTION AND ITS CODING IN SIMIAN NEOCORTEX
Aseyev, N. (2023) · Журнал высшей нервной деятельности им И П Павлова · doi:10.31857/s0044467723010045 · Cited by 1
Butorovа, A. S., Khomenko, N. M., Kuzmin, S. E., & Sergeev, A. (2025) · Ural Symposium on Biomedical Engineering, Radioelectronics and Information Technology (USBEREIT) · doi:10.1109/usbereit65494.2025.11054174 · Cited by 1
Re-imagining sensory substitution through gestural control: Point-To-Tell 2
Ruan, L., Hamilton-Fletcher, G., Beheshti, M., Hudson, T. E., Porfiri, M., & Rizzo, J. (2025) · Disability and Rehabilitation Assistive Technology · doi:10.1080/17483107.2025.2590679 · Cited by 2
“For example, research by Abboud et al. [2] on the EyeMusic SSD and subsequent work by Maidenbaum et al. [54] show that when users’ own motor signals are integrated into the feedback loop, perceptual learning is enhanced and object relationships become clearer.”
Ingco, M., & Yoon, S. (2026) · IEEE International Conference on Artificial Intelligence and eXtended and Virtual Reality (AIxVR) · doi:10.1109/aixvr67263.2026.00041 · Cited by 1
“Since vOICe systems scan image frames left-to-right column-wise, it takes about 1 to 2 seconds to scan a single image [16].”
The effectiveness of non-drug rehabilitation methods for hemianopia after a cerebrovascular accident
Sergeeva, M. S., Romanchuk, N., Komarova, Y., Zakharov, A. V., & Shirolapov, I. V. (2026) · S S Korsakov Journal of Neurology and Psychiatry · doi:10.17116/jnevro202612603252 · Cited by 1
Maidenbaum, S. (2015) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/2702613.2702626 · Cited by 2
“including 3 journal research papers, a high-impact review in NBR and 5 conference publications as lead author and several more as a contributing author (see references [1-7] and appendix 3 for some examples).”
BackgroundSelf-citationLocal Dissimilarity Measures of Frames in Visual Substitution System for Blind People
Jabnoun, H., Morain-Nicolier, F., Bensarti, F., & Amiri, H. (2016) · International Conference on Computer Graphics, Imaging and Visualization (CGIV) · doi:10.1109/cgiv.2016.53 · Cited by 1
ANAVI: Advanced Navigation Assistance for Visually Impaired
Sharma, A., Vasan, V. R., & Bharathi, S. P. (2020) · Advances in intelligent systems and computing · doi:10.1007/978-981-15-5566-4_25 · Cited by 1
Guo, X.-F., & Wang, S. (2020) · European Journal of Horticultural Science · doi:10.17660/ejhs.2020/85.6.2 · Cited by 1
Commère, L., & Rouat, J. (2022) · IEEE Transactions on Human-Machine Systems · doi:10.1109/thms.2022.3194715 · Cited by 3
Liang, J., Li, H., Chai, X., Gao, Q., Zhou, M., Guo, T., Chen, Y., & Di, L. (2024) · Journal of Neural Engineering · doi:10.1088/1741-2552/ad94a4 · Cited by 1
“Drawing inspiration from the foundational principles of the vOICe system, the Sami group from Israel conceived an auditory assistive device known as Eyemusic [42].”
Effectiveness Analysis of Structured Training Program for Sensory Substitution Systems
Park, Y., Moon, K.-D., Kim, M., Park, S. J., & Jeong, C. Y. (2025) · International Conference on Advanced Communications Technology (ICACT) · doi:10.23919/icact63878.2025.10936727 · Cited by 1
“Experiments using EyeMusic involved tasks where subjects distinguished colors or located objects of specific colors, assessing color recognition accuracy and object search speed [4].”
Ueda, S., Sakai, H., & Kumada, T. (2019) · Scientific Reports · doi:10.1038/s41598-019-54324-6 · Cited by 1
Virtual Self-Training of a Sensory Substitution Device for Blind Individuals
Buchs, G., Heimler, B., Kerem, M., Maidenbaum, S., Braun, L., & Amedi, A. (2019) · International Conference on Virtual Rehabilitation · doi:10.1109/icvr46560.2019.8994662 · Cited by 1
“Visual-to-auditory SSDs are a family of non-invasive devices that convert visual input to audition according to a specific algorithm [2], [3].”
BackgroundSelf-citationMahani, M.-A. N., Bausenhart, K. M., Ulrich, R., & Ahmadabadi, M. N. (2022) · IEEE Transactions on Cognitive and Developmental Systems · doi:10.1109/tcds.2022.3144092 · Cited by 1
“such as substitution systems, can partially compensate for the loss of a sensory modality [2], [3], while others are designed to improve perception by providing complementary or processed information [4], [5].”
BackgroundISEE: A Wearable Image-Sound Translation System for Partially Sighted People
Zhang, Z., Xiang, C., Zhao, Z., Liang, W., Cui, D., & Liu, H. (2023) · IEEE Sensors Journal · doi:10.1109/jsen.2023.3266793 · Cited by 3
“The EyeMusic system differs that the ordinate axis height is represented by musical notes [14].”
BackgroundColorophone 2.2 – A Spatial, Real-Time Color-to-Sound Sensory Substitution DevicePreprint
Osiński, D., Łukowska, M., Kałwak, W., Wierzchoń, M., & Hjelme, D. R. (2022) · doi:10.36227/techrxiv.21385140 · Cited by 1
Mühlinghaus, S., Schmidt, V., & Peter König, B. (2024) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2024.07.17.603884 · Cited by 1
“Examples include augmented and virtual reality (VR) in the visual domain (5, 6), the manipulation of the tactile domain (7), and also tactile augmentation of spatial cues (2, 3, 8) or the mapping of sounds to real-time data recorded by exter-nal sensors in the auditory domain (9, 10).”
BackgroundTechnologies for vision impairment
Ayton, L. N., Allen, P. J., Abbott, C. J., & Petoe, M. A. (2019) · Routledge eBooks · doi:10.4324/9781315111353-25
Mahani, M.-A. N., Bausenhart, K. M., Ulrich, R., & Ahmadabadi, M. N. (2020) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2020.01.29.924662
МЕТОД КОДУВАННЯ КОНТУРІВ МУЗИЧНИМ ЗВУКОМ
Vakal, L. P., & Вакал, Є. С. (2021) · Applied Questions of Mathematical Modeling · doi:10.32782/kntu2618-0340/2021.4.1.5 · Cited by 1
Can We Combine Senses to Create New “Super Senses” and Abilities?
Maimon, A., & Amedi, A. (2022) · Frontiers for Young Minds · doi:10.3389/frym.2022.1009105
Self-citationColorophone 2Preprint
Osiński, D., Łukowska, M., Kałwak, W., Wierzchoń, M., & Hjelme, D. R. (2022) · doi:10.36227/techrxiv.21385140.v1
Matsuda, S., Nakagawa, Y., Funazaki, Y., Matsuyama, N., Nakamura, S., Komatsu, T., Torii, T., Sumikawa, R., & Takao, H. (2022) · Procedia Computer Science · doi:10.1016/j.procs.2022.09.514
“Abboud et al. [6] proposed a method, “EyeMusic”, which uses musical tones generated from natural musical instruments to comfortably convey visual information to the visually impaired.”
MethodValdés, S. I. R. (2023) · UPM Digital Archive (Technical University of Madrid) · doi:10.20868/upm.thesis.76515 · Cited by 1
Wald, I. Y., & Maimon, A. (2025) · ACM SIGCHI Symposium on Engineering Interactive Computing Systems (EICS) · doi:10.1145/3731406.3731971
TaK: A Tactile Knob to Investigate Orientation Perception
Vitale, A., Bertolasi, J., Orciari, L., Lorini, C., Parmiggiani, A., Murray, M. M., Wallace, M. T., & Gori, M. (2025) · IEEE International Symposium on Medical Measurements and Applications · doi:10.1109/memea65319.2025.11068010
“For blind individuals, touch and audio become primary sources of information, compensating for the absence of vision [1], [2], [3].”
The Visual Echolocation System with an Integrated 3D CameraPreprint
Bizoń-Angov, P., Osiński, D., Wierzchoń, M., & Konieczny, J. (2025) · doi:10.36227/techrxiv.175756652.28904908/v1
Enhancing Emotional Congruence in Sensory Substitution
Cenerini, C., Vollero, L., Pennazza, G., Keller, F., & Çeliktutan, O. (2025) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2025.3627092 · Cited by 1
“WhilecurrentSSDshavemaderemarkablestridesincon-veyingspatialandtemporalinformation[3],[4],theylargely failtotranslatethe emotional essence of visual experiences.”
Adepoju, B., Yoo, S., & Nimon, K. (2026) · IGI Global eBooks · doi:10.4018/979-8-3693-3844-5.ch005 · Cited by 1
ThermalWave: A Wearable Thermal-to-Audio Sensory Substitution Device
Maimon, A., Wald, I. Y., & Lanir, J. (2026) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3772363.3798941
“While this may limit the achievable real-time perception, similar image-to-sound algorithms such as the vOICe, or EyeMusic [1, 35], have been used successfully in the past for standard images.”
Reprogramming Perception: A Neuroplasticity-Focused Design Space for Sensory Substitution Devices
Maimon, A., Malaka, R., & Wald, I. Y. (2026) · Augmented Humans International Conference (AHs) · doi:10.1145/3795011.3795025
Park, S., Bak, S., & Lee, H. (2026) · British Journal of Visual Impairment · doi:10.1177/02646196261450976
Gwiaździński, P., Reuter, M., Dubis, J., Osiński, D., & Wierzchoń, M. (2026) · Disability and Rehabilitation Assistive Technology · doi:10.1080/17483107.2026.2699832
Moon, K. D., Kyung, P. Y., Kim, M. S., & Jeong, C. Y. (2026) · Frontiers in Psychology · doi:10.3389/fpsyg.2026.1887888
Bilal, S., Bongers, R. M., & Jacobs, D. M. (2026) · Frontiers in Psychology · doi:10.3389/fpsyg.2026.1674806
Aligning sensations: Do trumpets sound scarlet?
Spence, C. J., & Stefano, N. D. (2026) · i-Perception · doi:10.1177/20416695261468649
“…or timbres is of increasing contemporary relevance, given the multitude of attempts to build digital sensory substitution systems for the blind (Abboud, Hanassy et al., 2014; Cavaco et al., 2013; Cho et al., 2020; Deville et al., 2009; Payling et al., 2007; see also Barbiere et al., 2007) (see…”
Metadata
- Authors
- Abboud, S., Hanassy, S., Levy-Tzedek, S., Maidenbaum, S., & Amedi, A.
- Affiliations
- Hebrew University of Jerusalem (IL)
- Department of Medical Neurobiology, The Institute for Medical Research Israel-Canada, Faculty of Medicine, The Hebrew University of Jerusalem, Jerusalem, Israel
- Corresponding author
- Amedi, Amir
- Venue
- Restorative Neurology and Neuroscience
- Publisher
- SAGE Publications
- Published
- 2014-01-01
- Volume
- 32
- Issue
- 2
- Pages
- 247-257
- ISSN
- 0922-6028 (ISSN-L)
- 1878-3627
- OpenAlex
- W1922152898
- Cited by
- 199
- References
- 43
- Citations by year
- 2026: 11 · 2025: 14 · 2024: 16 · 2023: 17 · 2022: 20 · 2021: 16 · 2020: 18 · 2019: 17 · 2018: 10 · 2017: 12 · 2016: 17 · 2015: 22 · 2014: 9
- FWCI
- 11.15
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Restorative Neurology and Neuroscience (publishedVersion, cc-by-nc)
- PubMed (publishedVersion)
- Funders
- James S. McDonnell Foundation (220020284)
- Israel Science Foundation (ISF 1684/08, 1684/08)
- FP7 Ideas: European Research Council (310809)
- Topics
- Life Sciences › Neuroscience › Cognitive Neuroscience › Tactile and Sensory Interactions
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Multisensory perception and integration
- Health Sciences › Health Professions › Occupational Therapy › Assistive Technology in Communication and Mobility
- Keywords
- sensory substitution devices, color perception, shape perception, blind users, pentatonic scale, visual rehabilitation
- MeSH terms
- Acoustic Stimulation, Adult, Auditory Perception, Blindness, Female, Humans, Male, Middle Aged, Neuronal Plasticity, Sensation, Persons with Visual Disabilities, Education of Persons with Visual Disabilities
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.