CoChat

Deep Learning-Based Optimization of Visual–Auditory Sensory Substitution

Open access
Kim, Mooseop, Park, Yunkyung, Moon, Kyeongdeok, Jeong, Chi Yoon · 2023 · IEEE Access

The DOI matches its Crossref record; 45 of the 45 references it cites were found in a registry or an index; 12 of the 38 sources its claims cite were checked for support.

Abstract

Visual–auditory sensory substitution systems can aid blind people in traveling to various places and recognizing their own environments without help from others. Although several such systems have been developed, they are either not widely used or are limited to laboratory-scale research. Among various factors that hinder the widespread use of these systems, one of the most important issues to consider is the optimization of the algorithms for sensory substitution. This study is the first attempt at exploring the possibility of using deep learning for the objective quantification of sensory substitution. To this end, we used generative adversarial networks to investigate the possibility of optimizing the vOICe algorithm, a representative visual–auditory sensory substitution method, by controlling the parameters of the method for converting an image to sound. Furthermore, we explored the effect of the parameters on the conversion scheme for the vOICe system and performed frequency-range and frequency-mapping-function experiments. The process of sensory substitution in humans was modeled to use generative models to assess the extent of visual perception from the substituted sensory signals. We verified the human-based experimental results against the modeling results. The results suggested that deep learning could be used for evaluating the efficiency of algorithms for visual–auditory sensory substitutions without labor-intensive human behavioral experiments. The introduction of deep learning for optimizing the visual–auditory conversion method is expected to facilitate studies on various aspects of sensory substitution, such as generalization and estimation of algorithm efficiency.

Overview

CoChat Summary

TL;DR

This paper introduces a GAN-based computational model to objectively evaluate and optimize visual–auditory sensory substitution, using the vOICe system as a test case. It finds that a wider frequency range and a proposed Mel-scale frequency mapping improve visual perception, with model-based results largely matching human behavioral experiments.

Key findings

  • A wide frequency range (80–7600 Hz) was more efficient than the conventional vOICe range (500–5000 Hz): the GAN model showed a significantly lower FID score for 80–7600 Hz, and human participants trained with the wide range showed a steady increase in visual perception, whereas the narrow-range group improved only moderately from session 3.
  • The proposed Mel-scale frequency mapping achieved the best visual perception in both model-based analysis and human behavioral experiments. In the human experiment, average increase rates were 10.2% for Mel-scale, 8.54% for Tanh, and 7.56% for Exponential mapping; Tanh showed a higher average increase rate than Exponential but the lowest overall performance.
  • The paper reports that deep learning-based analysis results were similar to human behavioral results, suggesting deep learning can evaluate visual–auditory conversion efficiency without labor-intensive human experiments.

Methods

  • The proposed computational model used vOICe for visual-to-auditory conversion, a CNN-based audio feature embedding network, and an auxiliary classifier GAN to generate images from converted audio; generated image quality was assessed with Fréchet inception distance (FID). The MNIST dataset was split into 60,000 training and 10,000 testing images, with 20% of training data used for validation.
  • Human experiments included a preliminary study with 10 sighted normal-hearing participants aged 20–27 years (male:female = 5:5, mean age = 22.3 ± 2.16 years) assigned to two frequency-range groups, and a main experiment with 30 volunteers aged 19–30 years (male:female = 16:14, mean age = 23.83 ± 2.83 years) randomly assigned to three frequency-mapping groups (n = 10 each).
  • Behavioral performance was measured by proportion of correct choices in four-alternative forced-choice tasks, with a chance level of 0.25.

Limitations

  • The paper states that COVID-19 made it difficult to recruit sufficient participants; it did not test usability for other SSD systems; it used only the vOICe system; participants were from a specific age group; human experiments used a few basic shapes in a well-controlled laboratory environment, not natural environments; and it did not thoroughly test a large number of parameters.
  • Evident from the text: the model was trained on MNIST handwritten digits rather than real-world images, and the human samples were small (10 in the preliminary study, 30 in the main experiment). The main experiment’s text does not state that blind or visually impaired participants were included; the preliminary study explicitly used sighted subjects.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1109/access.2023.3243641

Cites

45 of 45 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Verification

Claims the paper makes

9 Valid · 0 Not valid · 21 Not checked · 66 Own results and interpretations

23 more claims

Create a free account to see all 96 claims and every source verdict.

See all claims — free account
  • It may state two separate claims as one. 35
  • Probably not a substantial claim. 51
  • The quote only partly states this claim. 1

Create a free account to see these claims.

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

1 of 1

Citations

Cites

Cited by

See all 12

Metadata

Authors
Kim, M., Park, Y., Moon, K., & Jeong, C. Y.
Affiliations
  • Electronics and Telecommunications Research Institute (KR)
  • Artificial Intelligence Research Laboratory, Human Enhancement and Assistive Technology Research Section, Electronics Telecommunications Research Institute (ETRI), Daejeon, South Korea
Venue
IEEE Access
Published
2023
Volume
11
Pages
14169-14180
ISSN
2169-3536 (ISSN-L)
OpenAlex
W4319778127
Cited by
12
References
42
Citations by year
2026: 3 · 2025: 6 · 2024: 2 · 2023: 1
FWCI
1.80
Type
journal article
Language
en
Open access
Locations
  • IEEE Access (publishedVersion, cc-by)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
Funders
  • Electronics and Telecommunications Research Institute
  • Korean Government, Fundamental Technology Research for Human-Centric Autonomous Intelligent Systems (23ZS1200)
Topics
  • Life Sciences › Neuroscience › Cognitive Neuroscience › Tactile and Sensory Interactions
  • Social Sciences › Psychology › Experimental and Cognitive Psychology › Multisensory perception and integration
  • Health Sciences › Health Professions › Occupational Therapy › Assistive Technology in Communication and Mobility
Keywords
sensory substitution, voice, blind people, generative adversarial networks, frequency mapping, algorithm optimization

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-01, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-01 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.