CoChat

Analysis and Validation of Cross-Modal Generative Adversarial Network for Sensory Substitution

Open access
Kim, Mooseop, Park, YunKyung, Moon, KyeongDeok, Jeong, Chi Yoon · 2021 · International Journal of Environmental Research and Public Health

The DOI matches its Crossref record; 60 of the 60 references it cites were found in a registry or an index.

Abstract

Visual-auditory sensory substitution has demonstrated great potential to help visually impaired and blind groups to recognize objects and to perform basic navigational tasks. However, the high latency between visual information acquisition and auditory transduction may contribute to the lack of the successful adoption of such aid technologies in the blind community; thus far, substitution methods have remained only laboratory-scale research or pilot demonstrations. This high latency for data conversion leads to challenges in perceiving fast-moving objects or rapid environmental changes. To reduce this latency, prior analysis of auditory sensitivity is necessary. However, existing auditory sensitivity analyses are subjective because they were conducted using human behavioral analysis. Therefore, in this study, we propose a cross-modal generative adversarial network-based evaluation method to find an optimal auditory sensitivity to reduce transmission latency in visual-auditory sensory substitution, which is related to the perception of visual information. We further conducted a human-based assessment to evaluate the effectiveness of the proposed model-based analysis in human behavioral experiments. We conducted experiments with three participant groups, including sighted users (SU), congenitally blind (CB) and late-blind (LB) individuals. Experimental results from the proposed model showed that the temporal length of the auditory signal for sensory substitution could be reduced by 50%. This result indicates the possibility of improving the performance of the conventional vOICe method by up to two times. We confirmed that our experimental results are consistent with human assessment through behavioral experiments. Analyzing auditory sensitivity with deep learning models has the potential to improve the efficiency of sensory substitution.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.3390/ijerph18126216

Cites

60 of 60 references verified0 citation errors · 0 not checkable

Statistics

Figures

Figure 1 Overall architecture of the proposed computational model to imitate the cross-modal plasticity to perceive visual images through audio signal. A cross-modal generative adversarial network (GAN) learns an image-to-audio mapping using an image generator (GImage) and audio discriminator (DAudio). V2A encoding represents the vOICe, Rcomp is the compression ratio, and NAudio is the audio encoder.
Figure 1 Overall architecture of the proposed computational model to imitate the cross-modal plasticity to perceive visual images through audio signal. A cross-modal generative adversarial network (GAN) learns an image-to-audio mapping using an image generator (GImage) and audio discriminator (DAudio). V2A encoding represents the vOICe, Rcomp is the compression ratio, and NAudio is the audio encoder.CC BY · PubMed Central
Figure 2 Procedure for experiment on human evaluation. Group A, B, and C refer to groups of participants who trained with sounds encoded in 0.5, 1 and 2 s, respectively, through the vOICe. Each group consists of five participants with congenital blindness (CB) and late blindness (LB). Sighted users (SU) additionally participated in Group B for reference purposes and for comparison of the results of the experiment.
Figure 2 Procedure for experiment on human evaluation. Group A, B, and C refer to groups of participants who trained with sounds encoded in 0.5, 1 and 2 s, respectively, through the vOICe. Each group consists of five participants with congenital blindness (CB) and late blindness (LB). Sighted users (SU) additionally participated in Group B for reference purposes and for comparison of the results of the experiment.CC BY · PubMed Central
Figure 3 (A) The participant wearing a blindfold and listening to the soundscape produced from the shape image displayed on the screen, determining the mappings between shape and sound using the braille print by hand. (B) Example of braille print for session training. (C) Example of braille print for testing.
Figure 3 (A) The participant wearing a blindfold and listening to the soundscape produced from the shape image displayed on the screen, determining the mappings between shape and sound using the braille print by hand. (B) Example of braille print for session training. (C) Example of braille print for testing.CC BY · PubMed Central
Figure 4 Procedure for the preliminary experiment. Each group of compressed sound corresponded to a predefined compression ratio. In the experiment performed, we used six different compression ratios (n = 6).
Figure 4 Procedure for the preliminary experiment. Each group of compressed sound corresponded to a predefined compression ratio. In the experiment performed, we used six different compression ratios (n = 6).CC BY · PubMed Central
Figure 5 Mean performance obtained by three participant groups as a function of auditory sensitivity. The values of the bar charts indicate the average scores of the tests for each level. Error bars indicate standard errors of the mean.
Figure 5 Mean performance obtained by three participant groups as a function of auditory sensitivity. The values of the bar charts indicate the average scores of the tests for each level. Error bars indicate standard errors of the mean.CC BY · PubMed Central
Figure 6 Examples of generated images when different compression rates were used for the vOICe encoding. Each row shows a gradual change of each generated MNIST digit image with the change in compression ratio. The left-most column contains unencoded MNIST digits; the second left-most column contains digits encoded by the vOICe, but not compressed; columns from third left-most to the right contain digits encoded with compression ratios from 10% to 80%, increased by 10%.
Figure 6 Examples of generated images when different compression rates were used for the vOICe encoding. Each row shows a gradual change of each generated MNIST digit image with the change in compression ratio. The left-most column contains unencoded MNIST digits; the second left-most column contains digits encoded by the vOICe, but not compressed; columns from third left-most to the right contain digits encoded with compression ratios from 10% to 80%, increased by 10%.CC BY · PubMed Central
Figure 7 Performance of cross-modal GAN in terms of the (A) inception score (IS) and (B) Fréchet inception distance (FID) over the change of the compression ratio. The compression ratio for the experiment increased linearly from 10% to 90%.
Figure 7 Performance of cross-modal GAN in terms of the (A) inception score (IS) and (B) Fréchet inception distance (FID) over the change of the compression ratio. The compression ratio for the experiment increased linearly from 10% to 90%.CC BY · PubMed Central
Figure 8 Mean performance change in accuracy of the baseline classifier when the compression ratio for audio embedding increased linearly from 10% to 90%.
Figure 8 Mean performance change in accuracy of the baseline classifier when the compression ratio for audio embedding increased linearly from 10% to 90%.CC BY · PubMed Central
Figure 9 Mean performance and change in the training time (the upper row of the figure) and average proportion of correct answers (the lower row of the figure) obtained by three participant groups in training sessions. Both analyses were plotted against average behavioral score change across subject groups. (A) Group A, trained with short (0.5 s) soundscapes; (B) Group B, trained with normal (1 s) soundscapes; and (C) Group C, trained with long (2 s) soundscapes.
Figure 9 Mean performance and change in the training time (the upper row of the figure) and average proportion of correct answers (the lower row of the figure) obtained by three participant groups in training sessions. Both analyses were plotted against average behavioral score change across subject groups. (A) Group A, trained with short (0.5 s) soundscapes; (B) Group B, trained with normal (1 s) soundscapes; and (C) Group C, trained with long (2 s) soundscapes.CC BY · PubMed Central
Figure 10 The recognition performance of the crossover test with different length of sounds obtained from three groups of participants. (A) Group A, trained with short (0.5 s) soundscapes; (B) Group B, trained with normal (1 s) soundscapes; and (C) Group C, trained with long (2 s) soundscapes. The x-axis of each figure represents the temporal length of the soundscape used for training the other groups. The dotted lines in the figures represent the mean performance of each group after five training sessions.
Figure 10 The recognition performance of the crossover test with different length of sounds obtained from three groups of participants. (A) Group A, trained with short (0.5 s) soundscapes; (B) Group B, trained with normal (1 s) soundscapes; and (C) Group C, trained with long (2 s) soundscapes. The x-axis of each figure represents the temporal length of the soundscape used for training the other groups. The dotted lines in the figures represent the mean performance of each group after five training sessions.CC BY · PubMed Central

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 60 references

Research works printed but not linked: 2

  1. Lee, S.B., Jeong, C.Y., Moon, K., and Kim, C.K. (2019, January 15–16). A study on applicability of Autoencoder-based sensory substitution method using dynamic time warping. Proceedings of the Annual Fall Conference of Korean Multimedia Society, Seoul, Korea.
  2. Goodfellow, I. (2016). Nips 2016 tutorial: Generative adversarial networks. arXiv.

Cited by

See all 9

Metadata

Authors
Kim, M., Park, Y., Moon, K., & Jeong, C. Y.
Affiliations
  • Electronics and Telecommunications Research Institute (KR)
  • Human Enhancement & Assistive Technology Research Section, Artificial Intelligence Research Lab., Electronics Telecommunications Research Institute (ETRI), Daejeon 34129, Korea
Corresponding author
Jeong, Chi Yoon
Venue
International Journal of Environmental Research and Public Health
Publisher
MDPI AG
Published
2021-06-08
Volume
18
Issue
12
Pages
6216
ISSN
1660-4601 (ISSN-L)
OpenAlex
W3167811510
Cited by
8
References
58
Citations by year
2026: 2 · 2025: 1 · 2024: 1 · 2023: 2 · 2022: 2
FWCI
0.66
Type
journal article
Language
en
Open access
Locations
  • International Journal of Environmental Research and Public Health (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • RePEc: Research Papers in Economics (submittedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
Funders
Electronics and Telecommunications Research Institute (21ZS1200)
Topics
  • Life Sciences › Neuroscience › Cognitive Neuroscience › Tactile and Sensory Interactions
  • Social Sciences › Psychology › Experimental and Cognitive Psychology › Multisensory perception and integration
  • Health Sciences › Health Professions › Occupational Therapy › Assistive Technology in Communication and Mobility
Keywords
sensory substitution, auditory sensitivity, transmission latency, voice, blind and visually impaired users, assistive technology
MeSH terms
Acoustic Stimulation, Auditory Perception, Blindness, Humans, Vision, Ocular, Persons with Visual Disabilities

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Analysis and Validation of Cross-Modal Generative Adversarial Network for Sensory Substitution. https://app.cochat.ai/papers/10.3390/ijerph18126216
The paper
Kim, M., Park, Y., Moon, K., & Jeong, C. Y. (2021). Analysis and Validation of Cross-Modal Generative Adversarial Network for Sensory Substitution. International Journal of Environmental Research and Public Health. https://doi.org/10.3390/ijerph18126216

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.