Analysis and Validation of Cross-Modal Generative Adversarial Network for Sensory Substitution
Open accessThe DOI matches its Crossref record; 60 of the 60 references it cites were found in a registry or an index.
Abstract
Visual-auditory sensory substitution has demonstrated great potential to help visually impaired and blind groups to recognize objects and to perform basic navigational tasks. However, the high latency between visual information acquisition and auditory transduction may contribute to the lack of the successful adoption of such aid technologies in the blind community; thus far, substitution methods have remained only laboratory-scale research or pilot demonstrations. This high latency for data conversion leads to challenges in perceiving fast-moving objects or rapid environmental changes. To reduce this latency, prior analysis of auditory sensitivity is necessary. However, existing auditory sensitivity analyses are subjective because they were conducted using human behavioral analysis. Therefore, in this study, we propose a cross-modal generative adversarial network-based evaluation method to find an optimal auditory sensitivity to reduce transmission latency in visual-auditory sensory substitution, which is related to the perception of visual information. We further conducted a human-based assessment to evaluate the effectiveness of the proposed model-based analysis in human behavioral experiments. We conducted experiments with three participant groups, including sighted users (SU), congenitally blind (CB) and late-blind (LB) individuals. Experimental results from the proposed model showed that the temporal length of the auditory signal for sensory substitution could be reduced by 50%. This result indicates the possibility of improving the performance of the conventional vOICe method by up to two times. We confirmed that our experimental results are consistent with human assessment through behavioral experiments. Analyzing auditory sensitivity with deep learning models has the potential to improve the efficiency of sensory substitution.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures










Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“In [30], a cross-modal GANbased evaluation method was proposed to find the optimal auditory sensitivity to reduce the transmission latency in visual-auditory sensory substitution.”
From Deep Learning-Based Optimization of Visual–Auditory Sensory Substitution (2023) · cites this paper as [30]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Rethinking the Inception Architecture for Computer Vision
Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., & Wojna, Z. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.308 · Cited by 32,138
GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., & Hochreiter, S. (2017) · arXiv · doi:10.48550/arxiv.1706.08500 · Cited by 8,365
Improved Techniques for Training GANs
Salimans, T., Goodfellow, I., Zaremba, W., Cheung, V., Radford, A., & Chen, X. (2016) · arXiv · doi:10.48550/arxiv.1606.03498 · Cited by 6,965
(2011) · Cited by 2,752
Self-citationWord length and the structure of short-term memory
Baddeley, A. D., Thomson, N., & Buchanan, M. (1975) · Journal of Verbal Learning and Verbal Behavior · doi:10.1016/s0022-5371(75)80045-4 · Cited by 1,980
Conditional Image Synthesis With Auxiliary Classifier GANs
Odena, A., Olah, C., & Shlens, J. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1610.09585 · Cited by 1,837
Training Generative Adversarial Networks with Limited Data
Karras, T., Aittala, M., Hellsten, J., Laine, S., Lehtinen, J., & Aila, T. (2020) · arXiv · doi:10.48550/arxiv.2006.06676 · Cited by 1,170
Convolutional neural network: a review of models, methodologies and applications to object detection
Dhillon, A., & Verma, G. K. (2020) · Progress in Artificial Intelligence · doi:10.1007/s13748-019-00203-0 · Cited by 1,119 · cited 1×
“In the work of the DEEP-SEE [29] framework, computer vision algorithms and deep convolutional neural networks (CNNs) [30,31] were jointly exploited to detect, track, and recognize objects encountered during navigation in an outdoor environment in real time.”
MethodAn experimental system for auditory image representations
Meijer, P. B. L. (1992) · IEEE Transactions on Biomedical Engineering · doi:10.1109/10.121642 · Cited by 900
DRAW: A Recurrent Neural Network For Image Generation
Gregor, K., Danihelka, I., Graves, A., Rezende, D. J., & Wierstra, D. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1502.04623 · Cited by 841
Barratt, S., & Sharma, R. (2018) · arXiv · doi:10.48550/arxiv.1801.01973 · Cited by 732
Pitch discrimination in the early blind
Gougoux, F., Lepore, F., Lassonde, M., Voss, P., Zatorre, R. J., & Belin, P. (2004) · Nature · doi:10.1038/430309a · Cited by 422
Owens, A., Isola, P., McDermott, J., Torralba, A., Adelson, E. H., & Freeman, W. T. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.264 · Cited by 346
Striem-Amit, E., Cohen, L., Dehaene, S., & Amedi, A. (2012) · Neuron · doi:10.1016/j.neuron.2012.08.026 · Cited by 294
A Survey of Handwritten Character Recognition with MNIST and EMNIST
Baldominos, A., Saez, Y., & Isasi, P. (2019) · Applied Sciences · doi:10.3390/app9153169 · Cited by 279 · cited 1×
“We used the MNIST handwritten digits dataset [47] as the input image for our experiments because it has been used as a benchmark for most previous approaches.”
MethodAuvray, M., Hanneton, S., & O'Regan, J. K. (2007) · Perception · doi:10.1068/p5631 · Cited by 268
Visual experiences in the blind induced by an auditory sensory substitution device
Ward, J., & Meijer, P. (2010) · Consciousness and Cognition · doi:10.1016/j.concog.2009.10.006 · Cited by 247
Human augmentation: Past, present and future
Raisamo, R., Rakkolainen, I., Majaranta, P., Salminen, K., Rantala, J., & Farooq, A. (2019) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2019.05.008 · Cited by 236
Maidenbaum, S., Abboud, S., & Amedi, A. (2014) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2013.11.007 · Cited by 230
Capelle, C., Trullemans, C., Arno, P., & Veraart, C. (1998) · IEEE Transactions on Biomedical Engineering · doi:10.1109/10.720206 · Cited by 210
Abboud, S., Hanassy, S., Levy-Tzedek, S., Maidenbaum, S., & Amedi, A. (2014) · Restorative Neurology and Neuroscience · doi:10.3233/rnn-130338 · Cited by 199 · cited 1×
“[10] developed the EyeMusic system, which provided a musical image representation preserving the shape and integrating color information.”
BackgroundDeep Cross-Modal Audio-Visual Generation
Chen, L., Srivastava, S., Duan, Z., & Xu, C. (2017) · Proceedings of the on Thematic Workshops of ACM Multimedia 2017 · doi:10.1145/3126686.3126723 · Cited by 186
Loomis, J. M., Marston, J. R., Golledge, R. G., & Klatzky, R. L. (2005) · Journal of Visual Impairment & Blindness · doi:10.1177/0145482x0509900404 · Cited by 156 · cited 1×
“[11] sought to develop a guidance system to improve”
BackgroundLearn Keras for Deep Neural Networks: A Fast-Track Approach to Modern Deep Learning with Python
Moolayil, J. (2019) · Apress eBooks · doi:10.1007/978-1-4842-4240-7 · Cited by 153
Vertical sound localization in blind humans
Lewald, J. (2002) · Neuropsychologia · doi:10.1016/s0028-3932(02)00071-4 · Cited by 150
Convolutional neural network: Deep learning-based classification of building quality problems
Zhong, B., Xing, X., Love, P., Wang, X., & Luo, H. (2019) · Advanced Engineering Informatics · doi:10.1016/j.aei.2019.02.009 · Cited by 144
RENIER, L., COLLIGNON, O., POIRIER, C., TRANDUY, D., VANLIERDE, A., BOL, A., VERAART, C., & DEVOLDER, A. (2005) · NeuroImage · doi:10.1016/j.neuroimage.2005.01.047 · Cited by 128
Loomis, J. M., Marston, J. R., Golledge, R. G., et al. (2018) · PubMed · doi:10.1184/r1/6617567 · Cited by 125
Designing sensory-substitution devices: Principles, pitfalls and potential1
Kristjánsson, Á., Moldoveanu, A., Jóhannesson, Ó. I., Balan, O., Spagnol, S., Valgeirsdóttir, V. V., & Unnthorsson, R. (2016) · Restorative Neurology and Neuroscience · doi:10.3233/rnn-160647 · Cited by 116
From science to technology: Orientation and mobility in blind children and adults
Cuturi, L. F., Aggius-Vella, E., Campus, C., Parmiggiani, A., & Gori, M. (2016) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2016.08.019 · Cited by 107
A Recurrent Variational Autoencoder for Speech Enhancement
Leglaive, S., Alameda-Pineda, X., Laurent Girin, B., & Horaud, R. (2020) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp40776.2020.9053164 · Cited by 98
The Ponzo Illusion with Auditory Substitution of Vision in Sighted and Early-Blind Subjects
Renier, L., Laloyaux, C., Collignon, O., Tranduy, D., Vanlierde, A., Bruyer, R., & De Volder, A. G. (2005) · Perception · doi:10.1068/p5219 · Cited by 89
Tapu, R., Mocanu, B., & Zaharia, T. (2017) · Sensors · doi:10.3390/s17112473 · Cited by 87
CMCGAN: A Uniform Framework for Cross-Modal Visual-Audio Mutual Generation
Hao, W., Zhang, Z., & Guan, H. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.12329 · Cited by 79
Cronly-Dillon, J., Persaud, K., & Gregory, R. P. F. (1999) · Proceedings of the Royal Society of London. Series B: Biological Sciences · doi:10.1098/rspb.1999.0942 · Cited by 76
Kolarik, A. J., Scarfe, A. C., Moore, B. C. J., & Pardhan, S. (2017) · PLOS ONE · doi:10.1371/journal.pone.0175750 · Cited by 59
The Vibe: A Versatile Vision‐to‐Audition Sensory Substitution Device
Hanneton, S., Auvray, M., & Durette, B. (2010) · Applied Bionics and Biomechanics · doi:10.1080/11762322.2010.512734 · Cited by 57
Ricciardi, E., Handjaras, G., & Pietrini, P. (2014) · Experimental Biology and Medicine · doi:10.1177/1535370214538740 · Cited by 54
Brain (re)organization following visual loss
Voss, P. (2018) · Wiley Interdisciplinary Reviews Cognitive Science · doi:10.1002/wcs.1468 · Cited by 51
Jicol, C., Lloyd-Esenkaya, T., Proulx, M. J., Lange-Smith, S., Scheller, M., O'Neill, E., & Petrini, K. (2020) · Frontiers in Psychology · doi:10.3389/fpsyg.2020.01443 · Cited by 46
A framework to account for the effects of visual loss on human auditory abilities.
Kolarik, A. J., Pardhan, S., & Moore, B. C. J. (2021) · Psychological Review · doi:10.1037/rev0000279 · Cited by 46
Navigating from a Depth Image Converted into Sound
Stoll, C., Palluel-Germain, R., Fristot, V., Pellerin, D., Alleysson, D., & Graff, C. (2015) · Applied Bionics and Biomechanics · doi:10.1155/2015/543492 · Cited by 45
Visuo-auditory sensory substitution for mobility assistance: testing TheVIBE
Durette, B., Louveton, N., Alleysson, D., & Hérault, J. (2008) · HAL · Cited by 45
Navigation aid for blind persons by visual-to-auditory sensory substitution: A pilot study
Neugebauer, A., Rifai, K., Getzlaff, M., & Wahl, S. (2020) · PLOS ONE · doi:10.1371/journal.pone.0237344 · Cited by 41
Communication and Interaction With Semiautonomous Ground Vehicles by Force Control Steering
Martínez-García, M., Kalawsky, R. S., Gordon, T., Smith, T., Meng, Q., & Flemisch, F. O. (2020) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2020.3020217 · Cited by 40
Looktel—A comprehensive platform for computer-aided visual assistance
Sudol, J., Dialameh, O., Blanchard, C., & Dorcey, T. (2010) · 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition - Workshops · doi:10.1109/cvprw.2010.5543725 · Cited by 36
Proulx, M. J., Gwinnutt, J., Dell’Erba, S., Levy-Tzedek, S., de Sousa, A. A., & Brown, D. J. (2015) · Restorative Neurology and Neuroscience · doi:10.3233/rnn-150541 · Cited by 36
Label-preserving data augmentation for mobile sensor data
Kim, M., & Jeong, C. Y. (2021) · Multidimensional Systems and Signal Processing · doi:10.1007/s11045-020-00731-2 · Cited by 32
Self-citationA New Model of Human Steering Using Far-Point Error Perception and Multiplicative Control
Martínez-García, M., & Gordon, T. (2018) · Conference proceedings/Conference proceedings - IEEE International Conference on Systems, Man, and Cybernetics · doi:10.1109/smc.2018.00218 · Cited by 31
SoundSight: a mobile sensory substitution device that sonifies colour, distance, and temperature
Hamilton-Fletcher, G., Alvarez, J., Obrist, M., & Ward, J. (2022) · Journal on Multimodal User Interfaces · doi:10.1007/s12193-021-00376-w · Cited by 28
Hu, D., Wang, D., Li, X., Nie, F., & Wang, Q. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00816 · Cited by 21
Echoic Sensory Substitution Information in a Single Obstacle Circumvention Task
Kolarik, A. J., Scarfe, A. C., Moore, B. C. J., & Pardhan, S. (2016) · PLoS ONE · doi:10.1371/journal.pone.0160872 · Cited by 16
Spagnol, S., Baldan, S., & Unnþórsson, R. (2017) · IEEE International Workshop on Multimedia Signal Processing (MMSP) · doi:10.1109/mmsp.2017.8122220 · Cited by 13
Proulx, M. J., Brown, D. J., Esenkaya, T., Leveson, J. B., Todorov, O. S., Watson, S. H., & de Sousa, A. A. (2020) · Applied Ergonomics · doi:10.1016/j.apergo.2020.103072 · Cited by 12
Graulty, C. J., Papaioannou, O., Bauer, P., Pitts, M. A., & Canseco-Gonzalez, E. (2017) · Journal of Cognitive Neuroscience · doi:10.1162/jocn_a_01210 · Cited by 9
Functional MRI of Sensory Substitution in the Blind
Chan, K. C., Murphy, M. C., Bang, J. W., Sims, J. R., Kashkoush, J. I., & Nau, A. C. (2018) · Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC) · doi:10.1109/embc.2018.8513622 · Cited by 8
Autoencoding sensory substitution
Tóth, V., & Parkkonen, L. (2019) · Unpublished · doi:10.13140/rg.2.2.10576.87048 · Cited by 7 · cited 1×
“Recently, an implicit method for visual-auditory sensory substitution technology, called the autoencoded V2A (AEV2A) model [35], has been proposed.”
Method(2006) · Chalmers Publication Library (Chalmers University of Technology) · Cited by 6
SoundSight: A Mobile Sensory Substitution Device that Sonifies Colour, Distance, and TemperaturePreprint
Hamilton-Fletcher, G., Alvarez, J., Obrist, M., et al. (2019) · doi:10.31234/osf.io/xsjwr · Cited by 5
Uesaki, M., Ashida, H., Kitaoka, A., & Pasqualotto, A. (2019) · Scientific Reports · doi:10.1038/s41598-019-50912-8 · Cited by 4
Research works printed but not linked: 3
- Lee S.B. Jeong C.Y. Moon K. Kim C.K. A study on applicability of Autoencoder-based sensory substitution method using dynamic time warping Proceedings of the Annual Fall Conference of Korean Multimedia Society Seoul, Korea 15–16 November 2019
- Goodfellow I. Nips 2016 tutorial: Generative adversarial networks arXiv 2016 1701.00160
- Gulli A. Pal S. Deep Learning with Keras Packt Publishing Ltd. Birmingham, UK 2017 10.31234/osf.io/xsjwr · DOI
Cited by
Masal, K. M., Bhatlawande, S. S., & Shingade, S. D. (2023) · Multimedia Tools and Applications · doi:10.1007/s11042-023-16355-0 · Cited by 46
Salih, H. E. B., Takeda, K., Kobayashi, H., Kakizawa, T., Kawamoto, M., & Zempo, K. (2022) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph19063151 · Cited by 30
Augmenting perception: How artificial intelligence transforms sensory substitution
Longin, L., & Deroy, O. D. (2022) · Consciousness and Cognition · doi:10.1016/j.concog.2022.103280 · Cited by 22
Deep Learning-Based Optimization of Visual–Auditory Sensory Substitution
Kim, M., Park, Y., Moon, K., & Jeong, C. Y. (2023) · IEEE Access · doi:10.1109/access.2023.3243641 · Cited by 12
“Our approach is similar to that employed in [12] and [30] in that it uses vOICe as the encoding method for visual–auditory sensory substitution and creates synthesized visual data from the encoded audio signals by using a cross-modal GAN.”
MethodSelf-citationKim, M., Park, Y., Moon, K.-D., & Jeong, C. Y. (2024) · IEEE Access · doi:10.1109/access.2024.3419102 · Cited by 7
Self-citation
Metadata
- Authors
- Kim, M., Park, Y., Moon, K., & Jeong, C. Y.
- Affiliations
- Electronics and Telecommunications Research Institute (KR)
- Human Enhancement & Assistive Technology Research Section, Artificial Intelligence Research Lab., Electronics Telecommunications Research Institute (ETRI), Daejeon 34129, Korea
- Corresponding author
- Jeong, Chi Yoon
- Venue
- International Journal of Environmental Research and Public Health
- Publisher
- MDPI AG
- Published
- 2021-06-08
- Volume
- 18
- Issue
- 12
- Pages
- 6216
- ISSN
- 1660-4601 (ISSN-L)
- PMC
- PMC8228544
- OpenAlex
- W3167811510
- Cited by
- 8
- References
- 58
- Citations by year
- 2026: 2 · 2025: 1 · 2024: 1 · 2023: 2 · 2022: 2
- FWCI
- 0.66
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- International Journal of Environmental Research and Public Health (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
- Funders
- Electronics and Telecommunications Research Institute (21ZS1200)
- Topics
- Life Sciences › Neuroscience › Cognitive Neuroscience › Tactile and Sensory Interactions
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Multisensory perception and integration
- Health Sciences › Health Professions › Occupational Therapy › Assistive Technology in Communication and Mobility
- Keywords
- sensory substitution, auditory sensitivity, transmission latency, voice, blind and visually impaired users, assistive technology
- MeSH terms
- Acoustic Stimulation, Auditory Perception, Blindness, Humans, Vision, Ocular, Persons with Visual Disabilities
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.