Analysis and Validation of Cross-Modal Generative Adversarial Network for Sensory Substitution
Open accessThe DOI matches its Crossref record; 60 of the 60 references it cites were found in a registry or an index.
Abstract
Visual-auditory sensory substitution has demonstrated great potential to help visually impaired and blind groups to recognize objects and to perform basic navigational tasks. However, the high latency between visual information acquisition and auditory transduction may contribute to the lack of the successful adoption of such aid technologies in the blind community; thus far, substitution methods have remained only laboratory-scale research or pilot demonstrations. This high latency for data conversion leads to challenges in perceiving fast-moving objects or rapid environmental changes. To reduce this latency, prior analysis of auditory sensitivity is necessary. However, existing auditory sensitivity analyses are subjective because they were conducted using human behavioral analysis. Therefore, in this study, we propose a cross-modal generative adversarial network-based evaluation method to find an optimal auditory sensitivity to reduce transmission latency in visual-auditory sensory substitution, which is related to the perception of visual information. We further conducted a human-based assessment to evaluate the effectiveness of the proposed model-based analysis in human behavioral experiments. We conducted experiments with three participant groups, including sighted users (SU), congenitally blind (CB) and late-blind (LB) individuals. Experimental results from the proposed model showed that the temporal length of the auditory signal for sensory substitution could be reduced by 50%. This result indicates the possibility of improving the performance of the conventional vOICe method by up to two times. We confirmed that our experimental results are consistent with human assessment through behavioral experiments. Analyzing auditory sensitivity with deep learning models has the potential to improve the efficiency of sensory substitution.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures










Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“In [30], a cross-modal GANbased evaluation method was proposed to find the optimal auditory sensitivity to reduce the transmission latency in visual-auditory sensory substitution.”
From Deep Learning-Based Optimization of Visual–Auditory Sensory Substitution (2023) · cites this paper as [30]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Rethinking the Inception Architecture for Computer Vision
Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., & Wojna, Z. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.308 · Cited by 32,138
GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., & Hochreiter, S. (2017) · arXiv · doi:10.48550/arxiv.1706.08500 · Cited by 8,365
Improved Techniques for Training GANs
Salimans, T., Goodfellow, I., Zaremba, W., Cheung, V., Radford, A., & Chen, X. (2016) · arXiv · doi:10.48550/arxiv.1606.03498 · Cited by 6,965
(2011) · Cited by 2,752
Self-citationWord length and the structure of short-term memory
Baddeley, A. D., Thomson, N., & Buchanan, M. (1975) · Journal of Verbal Learning and Verbal Behavior · doi:10.1016/s0022-5371(75)80045-4 · Cited by 1,980
Conditional Image Synthesis With Auxiliary Classifier GANs
Odena, A., Olah, C., & Shlens, J. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1610.09585 · Cited by 1,837
Training Generative Adversarial Networks with Limited Data
Karras, T., Aittala, M., Hellsten, J., Laine, S., Lehtinen, J., & Aila, T. (2020) · arXiv · doi:10.48550/arxiv.2006.06676 · Cited by 1,170
Convolutional neural network: a review of models, methodologies and applications to object detection
Dhillon, A., & Verma, G. K. (2020) · Progress in Artificial Intelligence · doi:10.1007/s13748-019-00203-0 · Cited by 1,119 · cited 1×
“In the work of the DEEP-SEE [29] framework, computer vision algorithms and deep convolutional neural networks (CNNs) [30,31] were jointly exploited to detect, track, and recognize objects encountered during navigation in an outdoor environment in real time.”
MethodAn experimental system for auditory image representations
Meijer, P. B. L. (1992) · IEEE Transactions on Biomedical Engineering · doi:10.1109/10.121642 · Cited by 900
DRAW: A Recurrent Neural Network For Image Generation
Gregor, K., Danihelka, I., Graves, A., Rezende, D. J., & Wierstra, D. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1502.04623 · Cited by 841
Research works printed but not linked: 2
- Lee, S.B., Jeong, C.Y., Moon, K., and Kim, C.K. (2019, January 15–16). A study on applicability of Autoencoder-based sensory substitution method using dynamic time warping. Proceedings of the Annual Fall Conference of Korean Multimedia Society, Seoul, Korea.
- Goodfellow, I. (2016). Nips 2016 tutorial: Generative adversarial networks. arXiv.
Cited by
Masal, K. M., Bhatlawande, S. S., & Shingade, S. D. (2023) · Multimedia Tools and Applications · doi:10.1007/s11042-023-16355-0 · Cited by 45
Salih, H. E. B., Takeda, K., Kobayashi, H., Kakizawa, T., Kawamoto, M., & Zempo, K. (2022) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph19063151 · Cited by 30
Augmenting perception: How artificial intelligence transforms sensory substitution
Longin, L., & Deroy, O. D. (2022) · Consciousness and Cognition · doi:10.1016/j.concog.2022.103280 · Cited by 22
Deep Learning-Based Optimization of Visual–Auditory Sensory Substitution
Kim, M., Park, Y., Moon, K., & Jeong, C. Y. (2023) · IEEE Access · doi:10.1109/access.2023.3243641 · Cited by 12
“Our approach is similar to that employed in [12] and [30] in that it uses vOICe as the encoding method for visual–auditory sensory substitution and creates synthesized visual data from the encoded audio signals by using a cross-modal GAN.”
MethodSelf-citationKim, M., Park, Y., Moon, K.-D., & Jeong, C. Y. (2024) · IEEE Access · doi:10.1109/access.2024.3419102 · Cited by 7
Self-citation
Metadata
- Authors
- Kim, M., Park, Y., Moon, K., & Jeong, C. Y.
- Affiliations
- Electronics and Telecommunications Research Institute (KR)
- Human Enhancement & Assistive Technology Research Section, Artificial Intelligence Research Lab., Electronics Telecommunications Research Institute (ETRI), Daejeon 34129, Korea
- Corresponding author
- Jeong, Chi Yoon
- Venue
- International Journal of Environmental Research and Public Health
- Publisher
- MDPI AG
- Published
- 2021-06-08
- Volume
- 18
- Issue
- 12
- Pages
- 6216
- ISSN
- 1660-4601 (ISSN-L)
- PMC
- PMC8228544
- OpenAlex
- W3167811510
- Cited by
- 8
- References
- 58
- Citations by year
- 2026: 2 · 2025: 1 · 2024: 1 · 2023: 2 · 2022: 2
- FWCI
- 0.66
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- International Journal of Environmental Research and Public Health (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
- Funders
- Electronics and Telecommunications Research Institute (21ZS1200)
- Topics
- Life Sciences › Neuroscience › Cognitive Neuroscience › Tactile and Sensory Interactions
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Multisensory perception and integration
- Health Sciences › Health Professions › Occupational Therapy › Assistive Technology in Communication and Mobility
- Keywords
- sensory substitution, auditory sensitivity, transmission latency, voice, blind and visually impaired users, assistive technology
- MeSH terms
- Acoustic Stimulation, Auditory Perception, Blindness, Humans, Vision, Ocular, Persons with Visual Disabilities
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Analysis and Validation of Cross-Modal Generative Adversarial Network for Sensory Substitution. https://app.cochat.ai/papers/10.3390/ijerph18126216
- The paper
- Kim, M., Park, Y., Moon, K., & Jeong, C. Y. (2021). Analysis and Validation of Cross-Modal Generative Adversarial Network for Sensory Substitution. International Journal of Environmental Research and Public Health. https://doi.org/10.3390/ijerph18126216
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.