Deep Learning-Based Optimization of Visual–Auditory Sensory Substitution
Open accessThe DOI matches its Crossref record; 45 of the 45 references it cites were found in a registry or an index; 12 of the 38 sources its claims cite were checked for support.
Abstract
Visual–auditory sensory substitution systems can aid blind people in traveling to various places and recognizing their own environments without help from others. Although several such systems have been developed, they are either not widely used or are limited to laboratory-scale research. Among various factors that hinder the widespread use of these systems, one of the most important issues to consider is the optimization of the algorithms for sensory substitution. This study is the first attempt at exploring the possibility of using deep learning for the objective quantification of sensory substitution. To this end, we used generative adversarial networks to investigate the possibility of optimizing the vOICe algorithm, a representative visual–auditory sensory substitution method, by controlling the parameters of the method for converting an image to sound. Furthermore, we explored the effect of the parameters on the conversion scheme for the vOICe system and performed frequency-range and frequency-mapping-function experiments. The process of sensory substitution in humans was modeled to use generative models to assess the extent of visual perception from the substituted sensory signals. We verified the human-based experimental results against the modeling results. The results suggested that deep learning could be used for evaluating the efficiency of algorithms for visual–auditory sensory substitutions without labor-intensive human behavioral experiments. The introduction of deep learning for optimizing the visual–auditory conversion method is expected to facilitate studies on various aspects of sensory substitution, such as generalization and estimation of algorithm efficiency.
Overview
CoChat Summary
TL;DR
This paper introduces a GAN-based computational model to objectively evaluate and optimize visual–auditory sensory substitution, using the vOICe system as a test case. It finds that a wider frequency range and a proposed Mel-scale frequency mapping improve visual perception, with model-based results largely matching human behavioral experiments.
Key findings
- A wide frequency range (80–7600 Hz) was more efficient than the conventional vOICe range (500–5000 Hz): the GAN model showed a significantly lower FID score for 80–7600 Hz, and human participants trained with the wide range showed a steady increase in visual perception, whereas the narrow-range group improved only moderately from session 3.
- The proposed Mel-scale frequency mapping achieved the best visual perception in both model-based analysis and human behavioral experiments. In the human experiment, average increase rates were 10.2% for Mel-scale, 8.54% for Tanh, and 7.56% for Exponential mapping; Tanh showed a higher average increase rate than Exponential but the lowest overall performance.
- The paper reports that deep learning-based analysis results were similar to human behavioral results, suggesting deep learning can evaluate visual–auditory conversion efficiency without labor-intensive human experiments.
Methods
- The proposed computational model used vOICe for visual-to-auditory conversion, a CNN-based audio feature embedding network, and an auxiliary classifier GAN to generate images from converted audio; generated image quality was assessed with Fréchet inception distance (FID). The MNIST dataset was split into 60,000 training and 10,000 testing images, with 20% of training data used for validation.
- Human experiments included a preliminary study with 10 sighted normal-hearing participants aged 20–27 years (male:female = 5:5, mean age = 22.3 ± 2.16 years) assigned to two frequency-range groups, and a main experiment with 30 volunteers aged 19–30 years (male:female = 16:14, mean age = 23.83 ± 2.83 years) randomly assigned to three frequency-mapping groups (n = 10 each).
- Behavioral performance was measured by proportion of correct choices in four-alternative forced-choice tasks, with a chance level of 0.25.
Limitations
- The paper states that COVID-19 made it difficult to recruit sufficient participants; it did not test usability for other SSD systems; it used only the vOICe system; participants were from a specific age group; human experiments used a few basic shapes in a well-controlled laboratory environment, not natural environments; and it did not thoroughly test a large number of parameters.
- Evident from the text: the model was trained on MNIST handwritten digits rather than real-world images, and the human samples were small (10 in the preliminary study, 30 in the main experiment). The main experiment’s text does not state that blind or visually impaired participants were included; the preliminary study explicitly used sighted subjects.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
9 Valid · 0 Not valid · 21 Not checked · 66 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“However, this method makes it difficult for a participant to understand the conversion scheme because there is no consistent correlation between changes in the input image and the resulting sound obtained from AEV2A.”
§ Deep Learning-based Optimization of Visual-Auditory Sensory Substitution > C. DEEP LEARNING FOR VISUAL-AUDITORY CONVERSION
What the sources say
- Partly supportedcites [23] Tóth, 2019, Autoencoding sensory substitution, arXiv:1907.06286 sourceExperimental — accuracy not yet measured
Section 2.5.1 (page unavailable) says implicit conversion acts as a black box and that explicit methods are easier to grasp, supporting the difficulty of understanding AEV2A’s conversion scheme. It does not support the asserted absence of a consistent image–sound correlation: Section 4.1 reports measured correlations between visual and audio features, including r = .97 and r = .99. (LLM, conf=0.87)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Behavioral studies on visual-auditory SSDs investigated the level of task performance to estimate the progress of visual perception in specific goal-related behaviors such as object recognition [2], localization [8], and navigation [9].”
§ Deep Learning-based Optimization of Visual-Auditory Sensory Substitution > I. INTRODUCTION
What the sources say
- Supportedcites [2] source
Citation [8] is attached to localization, not to object recognition or navigation. In “Preliminary end user testing,” the paper reports a task in which users indicated an object’s location in 3D space across ten trials and reports their average localization error over the last five trials. This supports the attributed use of task performance to assess localization with a visual-to-auditory SSD. (LLM, conf=0.88)
- Supportedcites [8] source
The citation [8] is attached to localization, not object recognition or navigation. In §2.3.1, participants’ identification of object locations was scored for accuracy; §3.1 reports that location-identification errors fell during familiarization with the visual-to-auditory EyeMusic device. This supports the claim that behavioral task performance was used to assess progress in localization. (LLM, conf=0.92)
- Supportedcites [9] source
The citation [9] is attached to navigation, not to object recognition or localization. The Behavioral Assessment (sections 3.2 and 3.4) describes a navigation task and measures performance using travel time and navigation errors; the Discussion (section 5) reports improvement with practice. (LLM, conf=0.93)
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Although the images are not of a real-world environment, they can be an appropriate alternative considering that the main objective of this study was to explore the possibility of deep learning for the objective quantification of the visual perception of the substituted signal.”
§ Deep Learning-based Optimization of Visual-Auditory Sensory Substitution > Inspired by the outstanding ability of GAN to generate a new sample for cross-modal perception, we propose a new
What the sources say
- Partly supportedcites [31] sourceExperimental — accuracy not yet measured
The Introduction (p. 1) describes MNIST as handwritten-digit images widely used to test computer-vision and deep-learning techniques. It does not address whether those images are appropriate for objectively quantifying visual perception of a substituted signal; that application is the citing study’s own rationale. (LLM, conf=0.84)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“the FID score of Group A (80-7600 Hz) was significantly lower than that of Group B (500-5000 Hz).”
§ Deep Learning-based Optimization of Visual-Auditory Sensory Substitution > B. FREQUENCY RANGE OPTIMIZATION FOR VISUAL-AUDITORY SSD
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The behavioral results in Fig. 8 show that the proposed Melscaled frequency mapping achieved the best performance when compared with the existing conversion methods in terms of perceiving the visual information.”
§ Deep Learning-based Optimization of Visual-Auditory Sensory Substitution > B. FREQUENCY-MAPPING OPTIMIZATION
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
23 more claims
Create a free account to see all 96 claims and every source verdict.
- It may state two separate claims as one. 35
- Probably not a substantial claim. 51
- The quote only partly states this claim. 1
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Deep Learning-based Optimization of Visual-Auditory Sensory Substitution > A. FREQUENCY-RANGE OPTIMIZATION
“Therefore, our results show that converting visual information using the frequency range of 80-7600 Hz in SSDs is more efficient than that using 500-5500 Hz for perceiving visual information from audio signals.”
§ Deep Learning-based Optimization of Visual-Auditory Sensory Substitution > A. FREQUENCY-RANGE OPTIMIZATION
“The participants were randomly assigned to two experimental groups to train with different audio streams converted using different frequency ranges: Group A (G A , 80-7600 Hz) and Group B (G B , 500-5000 Hz).”
Citations
Cites
Conditional Generative Adversarial Nets
Mirza, M., & Osindero, S. (2014) · arXiv · doi:10.48550/arxiv.1411.1784 · Cited by 8,863 · cited 2×
“The auxiliary classifier GAN (ACGAN) [35] enhances the concept of CGAN [34] by using an additional auxiliary classifier in the discriminator to classify real and generated images.”
MethodImproved Techniques for Training GANs
Salimans, T., Goodfellow, I., Zaremba, W., Cheung, V., Radford, A., & Chen, X. (2016) · arXiv · doi:10.48550/arxiv.1606.03498 · Cited by 6,965
(2011) · Cited by 2,752
Conditional Image Synthesis With Auxiliary Classifier GANs
Odena, A., Olah, C., & Shlens, J. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1610.09585 · Cited by 1,837
A Review on Generative Adversarial Networks: Algorithms, Theory, and Applications
Gui, J., Sun, Z., Wen, Y., Tao, D., & Ye, J. (2023) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2021.3130191 · Cited by 1,231 · cited 1×
“Generative adversarial networks (GANs) [11] have gained interest as an important research topic in deep learning because they can generate synthetic samples that cannot be distinguished as true or false based on existing data.”
BackgroundAn experimental system for auditory image representations
Meijer, P. B. L. (1992) · IEEE Transactions on Biomedical Engineering · doi:10.1109/10.121642 · Cited by 900 · cited 6×
“…a given training session, the extent of perceived visual information of the group trained through Mel-scaled frequency mapping was the greatest, with an average increase rate of 10.2%, whereas the Exponential [15] and Tanh [12] mapping showed average increase rates of 7.56% and 8.54%, respectively.”
BackgroundMethodInfluentialOwens, A., Isola, P., McDermott, J., Torralba, A., Adelson, E. H., & Freeman, W. T. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.264 · Cited by 346
A Survey of Handwritten Character Recognition with MNIST and EMNIST
Baldominos, A., Saez, Y., & Isasi, P. (2019) · Applied Sciences · doi:10.3390/app9153169 · Cited by 279 · cited 1×
“Therefore, we used the MNIST handwritten digit dataset [31], which is publicly available, as the visual data for our experiment.”
MethodGelfand, S. A. (2016) · doi:10.1055/b-006-161125 · Cited by 239
Maidenbaum, S., Abboud, S., & Amedi, A. (2014) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2013.11.007 · Cited by 230
Safe Local Navigation for Visually Impaired Users With a Time-of-Flight and Haptic Feedback Device
Katzschmann, R. K., Araki, B., & Rus, D. (2018) · IEEE Transactions on Neural Systems and Rehabilitation Engineering · doi:10.1109/tnsre.2018.2800665 · Cited by 214 · cited 2×
“However, in visual–auditory substitution, various methods that can code visual information [15], [16], [17], [18], [23], [24], [25] are available.”
BackgroundMethodCapelle, C., Trullemans, C., Arno, P., & Veraart, C. (1998) · IEEE Transactions on Biomedical Engineering · doi:10.1109/10.720206 · Cited by 210 · cited 2×
“Prosthesis for substitution of vision by audition (PSVA) [16] does not use column-wise horizontal scans on visual images; instead, it uses frequency increments from the bottom to the top and from the left to the right of the image and a higher density of auditory pixels at the center of the image…”
BackgroundMethodAbboud, S., Hanassy, S., Levy-Tzedek, S., Maidenbaum, S., & Amedi, A. (2014) · Restorative Neurology and Neuroscience · doi:10.3233/rnn-130338 · Cited by 199 · cited 3×
“EyeMusic [17] allows the user to distinguish between five colors, whereas See ColOr [18] delivers seven colors.”
BackgroundMethodDeep Cross-Modal Audio-Visual Generation
Chen, L., Srivastava, S., Duan, Z., & Xu, C. (2017) · Proceedings of the on Thematic Workshops of ACM Multimedia 2017 · doi:10.1145/3126686.3126723 · Cited by 186 · cited 2×
“Like-wise, several studies have reported the use of deep learning to automatically convert one modality information into another [12], [13], [14].”
BackgroundLoomis, J. M., Marston, J. R., Golledge, R. G., & Klatzky, R. L. (2005) · Journal of Visual Impairment & Blindness · doi:10.1177/0145482x0509900404 · Cited by 156 · cited 1×
“According to an earlier study [29], virtual sounds are more efficient in providing guidance than semantic descriptions of speech because they can immediately convey the spatial information; this enables the efficient transfer of visual information as accurately as possible to participants.”
Background‘Visual’ Acuity of the Congenitally Blind Using Visual-to-Auditory Sensory Substitution
Striem-Amit, E., Guendelman, M., & Amedi, A. (2012) · PLoS ONE · doi:10.1371/journal.pone.0033136 · Cited by 117 · cited 1×
“…could enable visually impaired and partially sighted people to recognize colors or The associate editor coordinating the review of this manuscript and approving it for publication was Felix Albu . objects [1], [2], navigate and avoid obstacles [3], [4], and even identify facial expressions [5].”
BackgroundDesigning sensory-substitution devices: Principles, pitfalls and potential1
Kristjánsson, Á., Moldoveanu, A., Jóhannesson, Ó. I., Balan, O., Spagnol, S., Valgeirsdóttir, V. V., & Unnthorsson, R. (2016) · Restorative Neurology and Neuroscience · doi:10.3233/rnn-160647 · Cited by 116 · cited 1×
“According to previous studies [6], [7], many factors hinder the widespread use of SSDs: high cost, difficulty of use, difficulty of operation by users with visual impairment and blindness, and the long time required to adapt to these”
BackgroundCMCGAN: A Uniform Framework for Cross-Modal Visual-Audio Mutual Generation
Hao, W., Zhang, Z., & Guan, H. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.12329 · Cited by 79 · cited 1×
“Inspired by this study, the authors of [27] presented a unified model called cross-modal cycle GAN (CMC-GAN) to perform cross-modal mappings between image and audio.”
MethodFast, accurate reaching movements with a visual-to-auditory sensory substitution device
Levy-Tzedek, S., Hanassy, S., Abboud, S., Maidenbaum, S., & Amedi, A. (2012) · Restorative Neurology and Neuroscience · doi:10.3233/rnn-2012-110219 · Cited by 79 · cited 1×
“Behavioral studies on visual–auditory SSDs investigated the level of task performance to estimate the progress of visual perception in specific goal-related behaviors such as object recognition [2], localization [8], and navigation [9].”
BackgroundReview of Quantitative Empirical Evaluations of Technology for People with Visual Impairments
Brulé, E., Tomlinson, B. J., Metatla, O., Jouffrais, C., & Serrano, M. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376749 · Cited by 76 · cited 1×
“Many previous studies have investigated the perceptual experience of sensory substitution based on behavioral observations of the participants [19], [20], [21], [24], [25].”
BackgroundDynamic Crosswalk Scene Understanding for the Visually Impaired
Tian, S., Zheng, M., Zou, W., Li, X., & Zhang, L. (2021) · IEEE Transactions on Neural Systems and Rehabilitation Engineering · doi:10.1109/tnsre.2021.3096379 · Cited by 48 · cited 2×
“However, in visual–auditory substitution, various methods that can code visual information [15], [16], [17], [18], [23], [24], [25] are available.”
BackgroundMethodReal‐world multimodal lifelog dataset for human behavior study
Chung, S., Jeong, C. Y., Lim, J., Lim, J., Noh, K. J., Kim, G., & Jeong, H. (2021) · ETRI Journal · doi:10.4218/etrij.2020-0446 · Cited by 48 · cited 1×
“Recent advancements in deep learning have achieved remarkable success in various areas of research on artificial intelligence, thus attracting considerable attention in various domains [10].”
BackgroundSelf-citationNavigating from a Depth Image Converted into Sound
Stoll, C., Palluel-Germain, R., Fristot, V., Pellerin, D., Alleysson, D., & Graff, C. (2015) · Applied Bionics and Biomechanics · doi:10.1155/2015/543492 · Cited by 45 · cited 1×
“Behavioral studies on visual–auditory SSDs investigated the level of task performance to estimate the progress of visual perception in specific goal-related behaviors such as object recognition [2], localization [8], and navigation [9].”
BackgroundAuditory Sensory Substitution is Intuitive and Automatic with Texture Stimuli
Stiles, N. R. B., & Shimojo, S. (2015) · Scientific Reports · doi:10.1038/srep15628 · Cited by 45
Transforming 3D Coloured Pixels into Musical Instrument Notes for Vision Substitution Applications
Bologna, G., Deville, B., Pun, T., & Vinckenbosch, M. (2007) · EURASIP Journal on Image and Video Processing · doi:10.1155/2007/76204 · Cited by 42 · cited 2×
“EyeMusic [17] allows the user to distinguish between five colors, whereas See ColOr [18] delivers seven colors.”
BackgroundMethodNavigation aid for blind persons by visual-to-auditory sensory substitution: A pilot study
Neugebauer, A., Rifai, K., Getzlaff, M., & Wahl, S. (2020) · PLOS ONE · doi:10.1371/journal.pone.0237344 · Cited by 41 · cited 1×
“…could enable visually impaired and partially sighted people to recognize colors or The associate editor coordinating the review of this manuscript and approving it for publication was Felix Albu . objects [1], [2], navigate and avoid obstacles [3], [4], and even identify facial expressions [5].”
BackgroundLabel-preserving data augmentation for mobile sensor data
Kim, M., & Jeong, C. Y. (2021) · Multidimensional Systems and Signal Processing · doi:10.1007/s11045-020-00731-2 · Cited by 32 · cited 1×
“We used a traditional convolutional neural network (CNN) model [32] to extract the audio features; this model comprised four sequentially stacked convolutional blocks and three fully connected layers, where each convolutional block consisted of the convolutional, maxpooling, and activation layers.”
MethodSelf-citationSoundSight: a mobile sensory substitution device that sonifies colour, distance, and temperature
Hamilton-Fletcher, G., Alvarez, J., Obrist, M., & Ward, J. (2022) · Journal on Multimodal User Interfaces · doi:10.1007/s12193-021-00376-w · Cited by 28 · cited 2×
“…could enable visually impaired and partially sighted people to recognize colors or The associate editor coordinating the review of this manuscript and approving it for publication was Felix Albu . objects [1], [2], navigate and avoid obstacles [3], [4], and even identify facial expressions [5].”
BackgroundWang, N. Y.-H., Wang, H.-L. S., Wang, T.-W., Fu, S.-W., Lu, X., Wang, H.-M., & Tsao, Y. (2021) · IEEE Transactions on Neural Systems and Rehabilitation Engineering · doi:10.1109/tnsre.2020.3042655 · Cited by 24 · cited 1×
“Recently, Mel-spectrograms [37] have been widely used in machine learning tasks such as speech recognition, acoustic scene classification, and audio-based emotion recognition.”
MethodAudio description from image by modal translation network
Ning, H., Zheng, X., Yuan, Y., & Lu, X. (2021) · Neurocomputing · doi:10.1016/j.neucom.2020.10.053 · Cited by 23 · cited 1×
“In [28], the authors proposed a modal translation network (MT-Net) from visual to auditory senses to generate audio descriptions from images.”
BackgroundHu, D., Wang, D., Li, X., Nie, F., & Wang, Q. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00816 · Cited by 21 · cited 8×
“Like-wise, several studies have reported the use of deep learning to automatically convert one modality information into another [12], [13], [14].”
BackgroundMethodInfluentialA Sensory Substitution Device Inspired by the Human Visual System
Osinski, D., & Hjelme, D. R. (2018) · 2018 11th International Conference on Human System Interaction (HSI) · doi:10.1109/hsi.2018.8431078 · Cited by 15 · cited 1×
“…could enable visually impaired and partially sighted people to recognize colors or The associate editor coordinating the review of this manuscript and approving it for publication was Felix Albu . objects [1], [2], navigate and avoid obstacles [3], [4], and even identify facial expressions [5].”
BackgroundGuidelines for quantitative and qualitative studies of sensory substitution experience
Kałwak, W., Reuter, M., Łukowska, M., Majchrowicz, B., & Wierzchoń, M. (2018) · Adaptive Behavior · doi:10.1177/1059712318771690 · Cited by 14 · cited 2×
“study [22] argued that visual perception is not accompanied by a vision-like subjective experience even if a task that normally engages vision is accurately performed using the sensory substitution method.”
BackgroundMethodOsiński, D., Łukowska, M., Hjelme, D. R., & Wierzchoń, M. (2021) · Sensors · doi:10.3390/s21217351 · Cited by 11 · cited 1×
“…could enable visually impaired and partially sighted people to recognize colors or The associate editor coordinating the review of this manuscript and approving it for publication was Felix Albu . objects [1], [2], navigate and avoid obstacles [3], [4], and even identify facial expressions [5].”
BackgroundHamilton-Fletcher, G., & Chan, K. C. (2021) · Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC) · doi:10.1109/embc46164.2021.9630296 · Cited by 11 · cited 1×
“Many previous studies have investigated the perceptual experience of sensory substitution based on behavioral observations of the participants [19], [20], [21], [24], [25].”
BackgroundTraining-induced plasticity enables visualizing sounds with a visual-to-auditory conversion device
Lerousseau, J. P., Arnold, G., & Auvray, M. (2021) · Scientific Reports · doi:10.1038/s41598-021-94133-4 · Cited by 10 · cited 1×
“Many previous studies have investigated the perceptual experience of sensory substitution based on behavioral observations of the participants [19], [20], [21], [24], [25].”
BackgroundAnalysis and Validation of Cross-Modal Generative Adversarial Network for Sensory Substitution
Kim, M., Park, Y., Moon, K., & Jeong, C. Y. (2021) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph18126216 · Cited by 8 · cited 2×
“Our approach is similar to that employed in [12] and [30] in that it uses vOICe as the encoding method for visual–auditory sensory substitution and creates synthesized visual data from the encoded audio signals by using a cross-modal GAN.”
MethodSelf-citationAutoencoding sensory substitution
Tóth, V., & Parkkonen, L. (2019) · Unpublished · doi:10.13140/rg.2.2.10576.87048 · Cited by 7 · cited 2×
“However, in visual–auditory substitution, various methods that can code visual information [15], [16], [17], [18], [23], [24], [25] are available.”
MethodDeep Cross-Modal Audio-Visual GenerationPreprint
Chen, L., Srivastava, S., Duan, Z., & Xu, C. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1704.08292 · Cited by 7
‘Visual’ acuity of the congenitally blind using visual-to-auditory sensory substitution
Striem-Amit, E., Guendelman, M., & Amedi, A. (2012) · Seeing and Perceiving · doi:10.1163/187847612x648206 · Cited by 6
A model of co-saliency based audio attention
Zhao, X., Wang, X., & Cheng, D. (2020) · Multimedia Tools and Applications · doi:10.1007/s11042-020-09020-3 · Cited by 4 · cited 1×
“According to previous research [36], the audible frequency range of human hearing is 20 Hz–20 kHz; however, this range is not used for frequency mapping of vOICe.”
MethodMultidimensional Features Extraction Methods in Frequency Domain
Olivares-Mercado, J., Aguilar-Torres, G., Toscano-Medina, K., et al. (2012) · InTech eBooks · doi:10.5772/36704 · Cited by 2 · cited 1×
“…the frequency range used in the vOICe system to the Mel-scale frequency banks obtained by applying the widely used Mel-frequency transformation [38]. where M H and M L represent the center frequencies of the upper and lower filter banks of the Mel-scale, corresponding to the highest and lowest…”
MethodListen to the ImagePreprint
Hu, D., Wang, D., Li, X., Nie, F., & Wang, Q. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1904.09115 · Cited by 1
Autoencoding sensory substitution
Tóth, V., & Parkkonen, L. (2019) · arXiv · doi:10.48550/arxiv.1907.06286
Visualizing sounds: training-induced plasticity with a visual-to-auditory conversion device
Lerousseau, J. P., Arnold, G., & Auvray, M. (2021) · doi:10.1101/2021.01.14.426668
Cited by
Kim, M., Park, Y., Moon, K.-D., & Jeong, C. Y. (2024) · IEEE Access · doi:10.1109/access.2024.3419102 · Cited by 7
“The same researchers further utilized deep learning techniques to optimize the sensory substitution algorithm concerning frequency range and mapping function [14].”
MethodSelf-citationButorovа, A. S., & Sergeev, A. (2025) · The European Physical Journal Special Topics · doi:10.1140/epjs/s11734-025-01596-x · Cited by 3
“The computational model proposed by [83] consisted of two parts: an audio encoder that transformed images into auditory signals and extracted features from the transformed auditory data, and a GAN model.”
InfluentialMoon, K.-D., Park, Y. K., Kim, M., & Jeong, C. Y. (2025) · IEEE Transactions on Neural Systems and Rehabilitation Engineering · doi:10.1109/tnsre.2025.3548942 · Cited by 2
“With ongoing improvements in the utilization of multisensory organs [15], IT, and AI technologies [5], [16], the application scope of SS has expanded to various fields, including navigation, obstacle detection and avoidance, object recognition, object manipulation, and reading [11], [12], [17].”
Self-citationButorovа, A. S., Koryukin, E. A., Khomenko, N. M., & Sergeev, A. (2024) · Современные технологии в медицине · doi:10.17691/stm2024.16.4.03 · Cited by 3
Effectiveness Analysis of Structured Training Program for Sensory Substitution Systems
Park, Y., Moon, K.-D., Kim, M., Park, S. J., & Jeong, C. Y. (2025) · International Conference on Advanced Communications Technology (ICACT) · doi:10.23919/icact63878.2025.10936727 · Cited by 1
“Sixteen gray levels were used to represent intermediate values, and click sounds indicated the start and end of the converted sound [7][8].”
Self-citation
Metadata
- Authors
- Kim, M., Park, Y., Moon, K., & Jeong, C. Y.
- Affiliations
- Electronics and Telecommunications Research Institute (KR)
- Artificial Intelligence Research Laboratory, Human Enhancement and Assistive Technology Research Section, Electronics Telecommunications Research Institute (ETRI), Daejeon, South Korea
- Venue
- IEEE Access
- Published
- 2023
- Volume
- 11
- Pages
- 14169-14180
- ISSN
- 2169-3536 (ISSN-L)
- OpenAlex
- W4319778127
- Cited by
- 12
- References
- 42
- Citations by year
- 2026: 3 · 2025: 6 · 2024: 2 · 2023: 1
- FWCI
- 1.80
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Locations
- IEEE Access (publishedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- Electronics and Telecommunications Research Institute
- Korean Government, Fundamental Technology Research for Human-Centric Autonomous Intelligent Systems (23ZS1200)
- Topics
- Life Sciences › Neuroscience › Cognitive Neuroscience › Tactile and Sensory Interactions
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Multisensory perception and integration
- Health Sciences › Health Professions › Occupational Therapy › Assistive Technology in Communication and Mobility
- Keywords
- sensory substitution, voice, blind people, generative adversarial networks, frequency mapping, algorithm optimization
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-01, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-01 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.