Deep Learning-Based Optimization of Visual–Auditory Sensory Substitution
Open accessThe DOI matches its Crossref record; 46 of the 46 references it cites were found in a registry or an index; 12 of the 38 sources its claims cite were checked for support.
Abstract
Visual–auditory sensory substitution systems can aid blind people in traveling to various places and recognizing their own environments without help from others. Although several such systems have been developed, they are either not widely used or are limited to laboratory-scale research. Among various factors that hinder the widespread use of these systems, one of the most important issues to consider is the optimization of the algorithms for sensory substitution. This study is the first attempt at exploring the possibility of using deep learning for the objective quantification of sensory substitution. To this end, we used generative adversarial networks to investigate the possibility of optimizing the vOICe algorithm, a representative visual–auditory sensory substitution method, by controlling the parameters of the method for converting an image to sound. Furthermore, we explored the effect of the parameters on the conversion scheme for the vOICe system and performed frequency-range and frequency-mapping-function experiments. The process of sensory substitution in humans was modeled to use generative models to assess the extent of visual perception from the substituted sensory signals. We verified the human-based experimental results against the modeling results. The results suggested that deep learning could be used for evaluating the efficiency of algorithms for visual–auditory sensory substitutions without labor-intensive human behavioral experiments. The introduction of deep learning for optimizing the visual–auditory conversion method is expected to facilitate studies on various aspects of sensory substitution, such as generalization and estimation of algorithm efficiency.
Overview
CoChat Summary
TL;DR
This paper introduces a GAN-based computational model to objectively evaluate and optimize visual–auditory sensory substitution, using the vOICe system as a test case. It finds that a wider frequency range and a proposed Mel-scale frequency mapping improve visual perception, with model-based results largely matching human behavioral experiments.
Key findings
- A wide frequency range (80–7600 Hz) was more efficient than the conventional vOICe range (500–5000 Hz): the GAN model showed a significantly lower FID score for 80–7600 Hz, and human participants trained with the wide range showed a steady increase in visual perception, whereas the narrow-range group improved only moderately from session 3.
- The proposed Mel-scale frequency mapping achieved the best visual perception in both model-based analysis and human behavioral experiments. In the human experiment, average increase rates were 10.2% for Mel-scale, 8.54% for Tanh, and 7.56% for Exponential mapping; Tanh showed a higher average increase rate than Exponential but the lowest overall performance.
- The paper reports that deep learning-based analysis results were similar to human behavioral results, suggesting deep learning can evaluate visual–auditory conversion efficiency without labor-intensive human experiments.
Methods
- The proposed computational model used vOICe for visual-to-auditory conversion, a CNN-based audio feature embedding network, and an auxiliary classifier GAN to generate images from converted audio; generated image quality was assessed with Fréchet inception distance (FID). The MNIST dataset was split into 60,000 training and 10,000 testing images, with 20% of training data used for validation.
- Human experiments included a preliminary study with 10 sighted normal-hearing participants aged 20–27 years (male:female = 5:5, mean age = 22.3 ± 2.16 years) assigned to two frequency-range groups, and a main experiment with 30 volunteers aged 19–30 years (male:female = 16:14, mean age = 23.83 ± 2.83 years) randomly assigned to three frequency-mapping groups (n = 10 each).
- Behavioral performance was measured by proportion of correct choices in four-alternative forced-choice tasks, with a chance level of 0.25.
Limitations
- The paper states that COVID-19 made it difficult to recruit sufficient participants; it did not test usability for other SSD systems; it used only the vOICe system; participants were from a specific age group; human experiments used a few basic shapes in a well-controlled laboratory environment, not natural environments; and it did not thoroughly test a large number of parameters.
- Evident from the text: the model was trained on MNIST handwritten digits rather than real-world images, and the human samples were small (10 in the preliminary study, 30 in the main experiment). The main experiment’s text does not state that blind or visually impaired participants were included; the preliminary study explicitly used sighted subjects.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
9 Valid · 0 Not valid · 21 Not checked · 66 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“However, this method makes it difficult for a participant to understand the conversion scheme because there is no consistent correlation between changes in the input image and the resulting sound obtained from AEV2A.”
§ Deep Learning-based Optimization of Visual-Auditory Sensory Substitution > C. DEEP LEARNING FOR VISUAL-AUDITORY CONVERSION
What the sources say
- Partly supportedcites [23] Tóth, 2019, Autoencoding sensory substitution, arXiv:1907.06286 sourceExperimental — accuracy not yet measured
Section 2.5.1 (page unavailable) says implicit conversion acts as a black box and that explicit methods are easier to grasp, supporting the difficulty of understanding AEV2A’s conversion scheme. It does not support the asserted absence of a consistent image–sound correlation: Section 4.1 reports measured correlations between visual and audio features, including r = .97 and r = .99. (LLM, conf=0.87)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Behavioral studies on visual-auditory SSDs investigated the level of task performance to estimate the progress of visual perception in specific goal-related behaviors such as object recognition [2], localization [8], and navigation [9].”
§ Deep Learning-based Optimization of Visual-Auditory Sensory Substitution > I. INTRODUCTION
What the sources say
- Supportedcites [2] source
Citation [8] is attached to localization, not to object recognition or navigation. In “Preliminary end user testing,” the paper reports a task in which users indicated an object’s location in 3D space across ten trials and reports their average localization error over the last five trials. This supports the attributed use of task performance to assess localization with a visual-to-auditory SSD. (LLM, conf=0.88)
- Supportedcites [8] source
The citation [8] is attached to localization, not object recognition or navigation. In §2.3.1, participants’ identification of object locations was scored for accuracy; §3.1 reports that location-identification errors fell during familiarization with the visual-to-auditory EyeMusic device. This supports the claim that behavioral task performance was used to assess progress in localization. (LLM, conf=0.92)
- Supportedcites [9] source
The citation [9] is attached to navigation, not to object recognition or localization. The Behavioral Assessment (sections 3.2 and 3.4) describes a navigation task and measures performance using travel time and navigation errors; the Discussion (section 5) reports improvement with practice. (LLM, conf=0.93)
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Although the images are not of a real-world environment, they can be an appropriate alternative considering that the main objective of this study was to explore the possibility of deep learning for the objective quantification of the visual perception of the substituted signal.”
§ Deep Learning-based Optimization of Visual-Auditory Sensory Substitution > Inspired by the outstanding ability of GAN to generate a new sample for cross-modal perception, we propose a new
What the sources say
- Partly supportedcites [31] sourceExperimental — accuracy not yet measured
The Introduction (p. 1) describes MNIST as handwritten-digit images widely used to test computer-vision and deep-learning techniques. It does not address whether those images are appropriate for objectively quantifying visual perception of a substituted signal; that application is the citing study’s own rationale. (LLM, conf=0.84)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“the FID score of Group A (80-7600 Hz) was significantly lower than that of Group B (500-5000 Hz).”
§ Deep Learning-based Optimization of Visual-Auditory Sensory Substitution > B. FREQUENCY RANGE OPTIMIZATION FOR VISUAL-AUDITORY SSD
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The behavioral results in Fig. 8 show that the proposed Melscaled frequency mapping achieved the best performance when compared with the existing conversion methods in terms of perceiving the visual information.”
§ Deep Learning-based Optimization of Visual-Auditory Sensory Substitution > B. FREQUENCY-MAPPING OPTIMIZATION
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
23 more claims
Create a free account to see all 96 claims and every source verdict.
- It may state two separate claims as one. 35
- Probably not a substantial claim. 51
- The quote only partly states this claim. 1
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Deep Learning-based Optimization of Visual-Auditory Sensory Substitution > A. FREQUENCY-RANGE OPTIMIZATION
“Therefore, our results show that converting visual information using the frequency range of 80-7600 Hz in SSDs is more efficient than that using 500-5500 Hz for perceiving visual information from audio signals.”
§ Deep Learning-based Optimization of Visual-Auditory Sensory Substitution > A. FREQUENCY-RANGE OPTIMIZATION
“The participants were randomly assigned to two experimental groups to train with different audio streams converted using different frequency ranges: Group A (G A , 80-7600 Hz) and Group B (G B , 500-5000 Hz).”
Citations
Cites
Conditional Generative Adversarial Nets
Mirza, M., & Osindero, S. (2014) · arXiv · doi:10.48550/arxiv.1411.1784 · Cited by 8,863 · cited 2×
“The auxiliary classifier GAN (ACGAN) [35] enhances the concept of CGAN [34] by using an additional auxiliary classifier in the discriminator to classify real and generated images.”
MethodImproved Techniques for Training GANs
Salimans, T., Goodfellow, I., Zaremba, W., Cheung, V., Radford, A., & Chen, X. (2016) · arXiv · doi:10.48550/arxiv.1606.03498 · Cited by 6,965
(2011) · Cited by 2,752
Conditional Image Synthesis With Auxiliary Classifier GANs
Odena, A., Olah, C., & Shlens, J. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1610.09585 · Cited by 1,837
A Review on Generative Adversarial Networks: Algorithms, Theory, and Applications
Gui, J., Sun, Z., Wen, Y., Tao, D., & Ye, J. (2023) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2021.3130191 · Cited by 1,231 · cited 1×
“Generative adversarial networks (GANs) [11] have gained interest as an important research topic in deep learning because they can generate synthetic samples that cannot be distinguished as true or false based on existing data.”
BackgroundAn experimental system for auditory image representations
Meijer, P. B. L. (1992) · IEEE Transactions on Biomedical Engineering · doi:10.1109/10.121642 · Cited by 900 · cited 6×
“…a given training session, the extent of perceived visual information of the group trained through Mel-scaled frequency mapping was the greatest, with an average increase rate of 10.2%, whereas the Exponential [15] and Tanh [12] mapping showed average increase rates of 7.56% and 8.54%, respectively.”
BackgroundMethodInfluentialOwens, A., Isola, P., McDermott, J., Torralba, A., Adelson, E. H., & Freeman, W. T. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.264 · Cited by 346
A Survey of Handwritten Character Recognition with MNIST and EMNIST
Baldominos, A., Saez, Y., & Isasi, P. (2019) · Applied Sciences · doi:10.3390/app9153169 · Cited by 279 · cited 1×
“Therefore, we used the MNIST handwritten digit dataset [31], which is publicly available, as the visual data for our experiment.”
MethodGelfand, S. A. (2016) · doi:10.1055/b-006-161125 · Cited by 239
Maidenbaum, S., Abboud, S., & Amedi, A. (2014) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2013.11.007 · Cited by 230
Cited by
Kim, M., Park, Y., Moon, K.-D., & Jeong, C. Y. (2024) · IEEE Access · doi:10.1109/access.2024.3419102 · Cited by 7
“The same researchers further utilized deep learning techniques to optimize the sensory substitution algorithm concerning frequency range and mapping function [14].”
MethodSelf-citationButorovа, A. S., & Sergeev, A. (2025) · The European Physical Journal Special Topics · doi:10.1140/epjs/s11734-025-01596-x · Cited by 3
“The computational model proposed by [83] consisted of two parts: an audio encoder that transformed images into auditory signals and extracted features from the transformed auditory data, and a GAN model.”
InfluentialMoon, K.-D., Park, Y. K., Kim, M., & Jeong, C. Y. (2025) · IEEE Transactions on Neural Systems and Rehabilitation Engineering · doi:10.1109/tnsre.2025.3548942 · Cited by 2
“With ongoing improvements in the utilization of multisensory organs [15], IT, and AI technologies [5], [16], the application scope of SS has expanded to various fields, including navigation, obstacle detection and avoidance, object recognition, object manipulation, and reading [11], [12], [17].”
Self-citationButorovа, A. S., Koryukin, E. A., Khomenko, N. M., & Sergeev, A. (2024) · Современные технологии в медицине · doi:10.17691/stm2024.16.4.03 · Cited by 2
Effectiveness Analysis of Structured Training Program for Sensory Substitution Systems
Park, Y., Moon, K.-D., Kim, M., Park, S. J., & Jeong, C. Y. (2025) · International Conference on Advanced Communications Technology (ICACT) · doi:10.23919/icact63878.2025.10936727 · Cited by 1
“Sixteen gray levels were used to represent intermediate values, and click sounds indicated the start and end of the converted sound [7][8].”
Self-citation
Metadata
- Authors
- Kim, M., Park, Y., Moon, K., & Jeong, C. Y.
- Affiliations
- Electronics and Telecommunications Research Institute (KR)
- Artificial Intelligence Research Laboratory, Human Enhancement and Assistive Technology Research Section, Electronics Telecommunications Research Institute (ETRI), Daejeon, South Korea
- Venue
- IEEE Access
- Published
- 2023
- Volume
- 11
- Pages
- 14169-14180
- ISSN
- 2169-3536 (ISSN-L)
- OpenAlex
- W4319778127
- Cited by
- 12
- References
- 42
- Citations by year
- 2026: 3 · 2025: 6 · 2024: 2 · 2023: 1
- FWCI
- 1.80
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Locations
- IEEE Access (publishedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- Electronics and Telecommunications Research Institute
- Korean Government, Fundamental Technology Research for Human-Centric Autonomous Intelligent Systems (23ZS1200)
- Topics
- Life Sciences › Neuroscience › Cognitive Neuroscience › Tactile and Sensory Interactions
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Multisensory perception and integration
- Health Sciences › Health Professions › Occupational Therapy › Assistive Technology in Communication and Mobility
- Keywords
- sensory substitution, voice, blind people, generative adversarial networks, frequency mapping, algorithm optimization
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-01, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-01 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Deep Learning-Based Optimization of Visual–Auditory Sensory Substitution. https://app.cochat.ai/papers/10.1109/access.2023.3243641
- The paper
- Kim, M., Park, Y., Moon, K., & Jeong, C. Y. (2023). Deep Learning-Based Optimization of Visual–Auditory Sensory Substitution. IEEE Access. https://doi.org/10.1109/access.2023.3243641
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.