Stereosonic vision: Exploring visual-to-auditory sensory substitution mappings in an immersive virtual reality navigation paradigm
Open accessThe DOI matches its Crossref record; 21 of the 21 references it cites were found in a registry or an index.
Abstract
Sighted people predominantly use vision to navigate spaces, and sight loss has negative consequences for independent navigation and mobility. The recent proliferation of devices that can extract 3D spatial information from visual scenes opens up the possibility of using such mobility-relevant information to assist blind and visually impaired people by presenting this information through modalities other than vision. In this work, we present two new methods for encoding visual scenes using spatial audio: simulated echolocation and distance-dependent hum volume modulation. We implemented both methods in a virtual reality (VR) environment and tested them using a 3D motion-tracking device. This allowed participants to physically walk through virtual mobility scenarios, generating data on real locomotion behaviour. Blindfolded sighted participants completed two tasks: maze navigation and obstacle avoidance. Results were measured against a visual baseline in which participants performed the same two tasks without blindfolds. Task completion time, speed and number of collisions were used as indicators of successful navigation, with additional metrics exploring detailed dynamics of performance. In both tasks, participants were able to navigate using only audio information after minimal instruction. While participants were 65% slower using audio compared to the visual baseline, they reduced their audio navigation time by an average 21% over just 6 trials. Hum volume modulation proved over 20% faster than simulated echolocation in both mobility scenarios, and participants also showed the greatest improvement with this sonification method. Nevertheless, we do speculate that simulated echolocation remains worth exploring as it provides more spatial detail and could therefore be more useful in more complex environments. The fact that participants were intuitively able to successfully navigate space with two new visual-to-audio mappings for conveying spatial information motivates the further exploration of these and other mappings with the goal of assisting blind and visually impaired individuals with independent mobility.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
You Only Look Once: Unified, Real-Time Object Detection
Redmon, J., Divvala, S., Girshick, R., & Farhadi, A. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.91 · Cited by 53,803
KinectFusion: Real-time dense surface mapping and tracking
Newcombe, R. A., Fitzgibbon, A., Izadi, S., Hilliges, O., Molyneaux, D., Kim, D., Davison, A. J., Kohi, P., Shotton, J., & Hodges, S. (2011) · 2011 10th IEEE International Symposium on Mixed and Augmented Reality · doi:10.1109/ismar.2011.6092378 · Cited by 4,008 · cited 1×
“Extending this, recent years have seen the proliferation of methods for the fusion of global 3D maps of environments [65, 66, 105–107].”
Flaxman, S. R., Bourne, R. R. A., Resnikoff, S., Ackland, P., Braithwaite, T., Cicinelli, M. V., Das, A., Jonas, J. B., Keeffe, J., Kempen, J. H., et al. (2017) · The Lancet Global Health · doi:10.1016/s2214-109x(17)30393-5 · Cited by 3,477
Vision substitution by tactile image projection
BACH-Y-RITA, P., COLLINS, C. C., SAUNDERS, F. A., WHITE, B., & SCADDEN, L. (1969) · Nature · doi:10.1038/221963a0 · Cited by 956 · cited 2×
“The first SSDs were pioneered by Bach-y-Rita in the 1960s with his development of the Tactile Vision Sensory Substitution (TVSS) device [37]: subjects received vibrating patterns via an array of pins mounted on their backs and were able to differentiate between oriented parallel lines, simple…”
BackgroundMethodAn experimental system for auditory image representations
Meijer, P. B. L. (1992) · IEEE Transactions on Biomedical Engineering · doi:10.1109/10.121642 · Cited by 900
3-D sound for virtual reality and multimedia
Madole, D., & Begault, D. (1995) · Computer Music Journal · doi:10.2307/3680997 · Cited by 868
Pitch discrimination in the early blind
Gougoux, F., Lepore, F., Lassonde, M., Voss, P., Zatorre, R. J., & Belin, P. (2004) · Nature · doi:10.1038/430309a · Cited by 422
Navigation system for the blind: Auditory display modes and guidance
Loomis, J. M., Golledge, R. G., & Klatzky, R. L. (1998) · Presence: Teleoperators and Virtual Environments · doi:10.1162/105474698565677 · Cited by 366
Geography and the disabled: a survey with special reference to vision impaired and blind populations
Golledge, R. G. (1993) · Transactions of the Institute of British Geographers · doi:10.2307/623069 · Cited by 302
A summary of research investigating echolocation abilities of blind and sighted humans
Kolarik, A. J., Cirstea, S., Pardhan, S., & Moore, B. C. J. (2014) · Hearing Research · doi:10.1016/j.heares.2014.01.010 · Cited by 270
Cited by
SoundSpaces: Audio-Visual Navigation in 3D Environments
Chen, C., Jain, U., Schissler, C., Gari, S. V. A., Al-Halah, Z., Ithapu, V. K., Robinson, P., & Grauman, K. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58539-6_2 · Cited by 202
Navigation and perception of spatial layout in virtual echo-acoustic space
Dodsworth, C., Norman, L. J., & Thaler, L. (2020) · Cognition · doi:10.1016/j.cognition.2020.104185 · Cited by 67
Autoencoding sensory substitution
Tóth, V., & Parkkonen, L. (2019) · Unpublished · doi:10.13140/rg.2.2.10576.87048 · Cited by 7
“However, as such virtual environments lack tactile feedback, V2A training needs to resort solely to simplistic shapes and surroundings, such as walls and cylinder obstacles [150].”
BackgroundMethod
Metadata
- Authors
- Massiceti, D., Hicks, S. L., & van Rheede, J. J.
- Venue
- PLOS ONE
- Publisher
- Public Library of Science (PLoS)
- Published
- 2018-07-05
- Volume
- 13
- Issue
- 7
- Pages
- e0199389
- ISSN
- 1932-6203 (ISSN-L)
- OpenAlex
- W2819222535
- Cited by
- 54
- References
- 21
- FWCI
- 3.04
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Stereosonic vision: Exploring visual-to-auditory sensory substitution mappings in an immersive virtual reality navigation paradigm. https://app.cochat.ai/papers/10.1371/journal.pone.0199389
- The paper
- Massiceti, D., Hicks, S. L., & van Rheede, J. J. (2018). Stereosonic vision: Exploring visual-to-auditory sensory substitution mappings in an immersive virtual reality navigation paradigm. PLOS ONE. https://doi.org/10.1371/journal.pone.0199389
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.