Seeing with Sound? Exploring Different Characteristics of a Visual-to-Auditory Sensory Substitution Device
Open accessThe DOI matches its Crossref record; 13 of the 13 references it cites were found in a registry or an index.
Abstract
Sensory substitution devices convert live visual images into auditory signals, for example with a web camera (to record the images), a computer (to perform the conversion) and headphones (to listen to the sounds). In a series of three experiments, the performance of one such device (‘The vOICe’) was assessed under various conditions on blindfolded sighted participants. The main task that we used involved identifying and locating objects placed on a table by holding a webcam (like a flashlight) or wearing it on the head (like a miner's light). Identifying objects on a table was easier with a hand-held device, but locating the objects was easier with a head-mounted device. Brightness converted into loudness was less effective than the reverse contrast (dark being loud), suggesting that performance under these conditions (natural indoor lighting, novice users) is related more to the properties of the auditory signal (ie the amount of noise in it) than the cross-modal association between loudness and brightness. Individual differences in musical memory (detecting pitch changes in two sequences of notes) was related to the time taken to identify or recognise objects, but individual differences in self-reported vividness of visual imagery did not reliably predict performance across the experiments. In general, the results suggest that the auditory characteristics of the device may be more important for initial learning than visual associations.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
A sensorimotor account of vision and visual consciousness
O'Regan, J. K., & Noë, A. (2001) · Behavioral and Brain Sciences · doi:10.1017/s0140525x01000115 · Cited by 3,126 · cited 1×
“They interpreted their results in terms of sensorimotor theories of vision (eg O'Regan and Noe 2001).”
BackgroundVISUAL IMAGERY DIFFERENCES IN THE RECALL OF PICTURES
MARKS, D. F. (1973) · British Journal of Psychology · doi:10.1111/j.2044-8295.1973.tb01322.x · Cited by 1,778
Vision substitution by tactile image projection
BACH-Y-RITA, P., COLLINS, C. C., SAUNDERS, F. A., WHITE, B., & SCADDEN, L. (1969) · Nature · doi:10.1038/221963a0 · Cited by 956
The Handbook of Multisensory Processes
(2004) · doi:10.7551/mitpress/3422.001.0001 · Cited by 948
An experimental system for auditory image representations
Meijer, P. B. L. (1992) · IEEE Transactions on Biomedical Engineering · doi:10.1109/10.121642 · Cited by 900
Sensory substitution and the human-machine interface
Bach-y-Rita, P., & W. Kercel, S. (2003) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2003.10.013 · Cited by 858
Shape conveyed by visual-to-auditory sensory substitution activates the lateral occipital complex
Amedi, A., Stern, W. M., Camprodon, J. A., Bermpohl, F., Merabet, L., Rotman, S., Hemond, C., Meijer, P., & Pascual-Leone, A. (2007) · Nature Neuroscience · doi:10.1038/nn1912 · Cited by 451 · cited 3×
“Amedi et al (2007) trained sighted participants to use the device over several weeks (40 h of training) and examined their brain activity, using fMRI, when listening to the soundscapes relative to scrambled sounds or environmental sounds (eg `moo').”
BackgroundBrain Plasticity: ‘Visual’ Acuity of Blind Persons via the Tongue
Sampaio, E., Maris, S., & Bach-y-Rita, P. (2001) · Brain Research · doi:10.1016/s0006-8993(01)02667-1 · Cited by 258 · cited 1×
“Also, whilst blind participants may be expected to perform better using these devices, they do not necessarily perform qualitatively differently (eg Amedi et al 2007; Sampaio et al 2001).”
Background"Metaphorical" Mapping in Human Infants
Wagner, S., Winner, E., Cicchetti, D., & Gardner, H. (1981) · Child Development · doi:10.2307/1129200 · Cited by 179 · cited 1×
“Similar effects have been reported in infants which suggests that this is not a culturally or linguistically based phenomenon (Wagner et al 1981).”
BackgroundArno, P., De Volder, A. G., Vanlierde, A., Wanet-Defalque, M.-C., Streel, E., Robert, A., Sanabria-Bohórquez, S., & Veraart, C. (2001) · NeuroImage · doi:10.1006/nimg.2000.0731 · Cited by 148
Cited by
Abboud, S., Hanassy, S., Levy-Tzedek, S., Maidenbaum, S., & Amedi, A. (2014) · Restorative Neurology and Neuroscience · doi:10.3233/rnn-130338 · Cited by 199
“Various psychophysical experiments have demonstrated the ability of blind users to perform object and letter identification and discrimination (Auvray et al., 2007; Brown et al., 2011; Striem-Amit et al., 2012a), recognize patterns (Poirier et al.”
Multisensory perceptual learning and sensory substitution
Proulx, M. J., Brown, D. J., Pasqualotto, A., & Meijer, P. (2014) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2012.11.017 · Cited by 148
Self-citationSensory substitution as an artificially acquired synaesthesia
Ward, J., & Wright, T. (2014) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2012.07.007 · Cited by 94
Self-citationKlatzky, R. L., Giudice, N. A., Bennett, C. R., & Loomis, J. M. (2014) · Multisensory Research · doi:10.1163/22134808-00002447 · Cited by 74
How well do you see what you hear? The acuity of visual-to-auditory sensory substitution
Haigh, A., Brown, D. J., Meijer, P., & Proulx, M. J. (2013) · Frontiers in Psychology · doi:10.3389/fpsyg.2013.00330 · Cited by 72
Auditory Sensory Substitution is Intuitive and Automatic with Texture Stimuli
Stiles, N. R. B., & Shimojo, S. (2015) · Scientific Reports · doi:10.1038/srep15628 · Cited by 45
Navigation aid for blind persons by visual-to-auditory sensory substitution: A pilot study
Neugebauer, A., Rifai, K., Getzlaff, M., & Wahl, S. (2020) · PLOS ONE · doi:10.1371/journal.pone.0237344 · Cited by 41
Proulx, M. J., Gwinnutt, J., Dell’Erba, S., Levy-Tzedek, S., de Sousa, A. A., & Brown, D. J. (2015) · Restorative Neurology and Neuroscience · doi:10.3233/rnn-150541 · Cited by 36
“A more recent study with446 the TDU (Chebat, Schneider, Kupers, & Ptito, 2011)447 used a large scale navigation task in a corridor with448 obstacles, similar to the navigation employed with The449 vOICe (Brown et al., 2011).”
BackgroundMethodInfluentialSelf-citationThe role of visual deprivation and experience on the performance of sensory substitution devices
Stronks, H. C., Nau, A. C., Ibbotson, M. R., & Barnes, N. (2015) · Brain Research · doi:10.1016/j.brainres.2015.06.033 · Cited by 31
Visual objects in the auditory system in sensory substitution: How much information do we need?
Brown, D. J., Simpson, A. J. R., & Proulx, M. J. (2014) · Multisensory Research · doi:10.1163/22134808-00002462 · Cited by 23
Self-citationHu, D., Wang, D., Li, X., Nie, F., & Wang, Q. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00816 · Cited by 21
Spatial navigation with horizontally spatialized sounds in early and late blind individuals
Paré, S., Bleau, M., Djerourou, I., Malotaux, V., Kupers, R., & Ptito, M. (2021) · PLOS ONE · doi:10.1371/journal.pone.0247448 · Cited by 20
Guidelines for quantitative and qualitative studies of sensory substitution experience
Kałwak, W., Reuter, M., Łukowska, M., Majchrowicz, B., & Wierzchoń, M. (2018) · Adaptive Behavior · doi:10.1177/1059712318771690 · Cited by 14
Metadata
- Authors
- Brown, D., Macpherson, T., & Ward, J.
- Venue
- Perception
- Publisher
- SAGE Publications
- Published
- 2011-01-01
- Volume
- 40
- Issue
- 9
- Pages
- 1120-1135
- ISSN
- 0301-0066 (ISSN-L)
- 1468-4233
- OpenAlex
- W2089765964
- Cited by
- 72
- References
- 13
- FWCI
- 1.96
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.