Audio–Vision Substitution for Blind Individuals: Addressing Human Information Processing Capacity Limitations
Open accessThe DOI matches its Crossref record; 20 of the 20 references it cites were found in a registry or an index.
Abstract
In this contribution, we consider the factors that influence the information processing capacity of the person using sensory substitution devices, and the influence of how the translated information, here in audio, impacts performance. First, we review aspects of vision substitution by tactile and audio devices, and then we review key theory in human information processing limitations to devise and test use of an audio-vision substitution device, The vOICe, for recognizing visual objects with audio substitution for vision. Participants heard sonifications of two-dimensional (2-D) images and had to match them to alternatives presented either in visual or tactile modalities. To assess whether capacity limits constrain performance, objects were either presented with all information simultaneously (top and bottom as whole objects), or successively (top and bottom of the object one after the other). Performance was superior in the successive trials, indicative of a capacity limit in processing the auditory information. We discuss the implications for training protocols and design to provide a useful accessibility device for blind individuals.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Dual-task interference in simple tasks: data and theory
Pashler, H. (1994) · Psychological Bulletin · doi:10.1037/0033-2909.116.2.220 · Cited by 3,112
Pashler, H. (1997) · doi:10.7551/mitpress/5677.001.0001 · Cited by 1,607
An experimental system for auditory image representations
Meijer, P. B. L. (1992) · IEEE Transactions on Biomedical Engineering · doi:10.1109/10.121642 · Cited by 900
Perceptual load as a major determinant of the locus of selection in visual attention
Lavie, N., & Tsal, Y. (1994) · Perception & Psychophysics · doi:10.3758/bf03213897 · Cited by 870 · cited 1×
“As empirical support was found for both models [50], [53], [54], Lavie proposed that the contrasting results on the locus of selection could be explained by perceptual load [55], [56].”
ResultSubretinal electronic chips allow blind patients to read letters and combine them to words
Zrenner, E., Bartz-Schmidt, K. U., Benav, H., Besch, D., Bruckmann, A., Gabel, V.-P., Gekeler, F., Greppmaier, U., Harscher, A., Kibbel, S., et al. (2011) · Proceedings of the Royal Society B: Biological Sciences · doi:10.1098/rspb.2010.1747 · Cited by 819 · cited 1×
“Invasive techniques such as implants provide low resolution imagery by stimulating surviving retinal cells [2]–[5], cortex [6]–[9] or optic nerve [10].”
BackgroundSchmidt, E. M., Bak, M. J., Hambrecht, F. T., Kufta, C. V., O'Rourke, D. K., & Vallabhanath, P. (1996) · Brain · doi:10.1093/brain/119.2.507 · Cited by 672
Dobelle, W. H., Mladejovsky, M. G., & Girvin, J. P. (1974) · Science · doi:10.1126/science.183.4123.440 · Cited by 440
Ptito, M. (2005) · Brain · doi:10.1093/brain/awh380 · Cited by 310
Pattern recognition with the optic nerve visual prosthesis
Veraart, C., Wanet‐Defalque, M., Gérard, B., Vanlierde, A., & Delbeke, J. (2003) · Artificial Organs · doi:10.1046/j.1525-1594.2003.07305.x · Cited by 259
Noorsal, E., Sooksood, K., Xu, H., Hornig, R., Becker, J., & Ortmanns, M. (2012) · IEEE Journal of Solid-State Circuits · doi:10.1109/jssc.2011.2164667 · Cited by 212 · cited 1×
“Invasive techniques such as implants provide low resolution imagery by stimulating surviving retinal cells [2]–[5], cortex [6]–[9] or optic nerve [10].”
BackgroundCapelle, C., Trullemans, C., Arno, P., & Veraart, C. (1998) · IEEE Transactions on Biomedical Engineering · doi:10.1109/10.720206 · Cited by 210
Abboud, S., Hanassy, S., Levy-Tzedek, S., Maidenbaum, S., & Amedi, A. (2014) · Restorative Neurology and Neuroscience · doi:10.3233/rnn-130338 · Cited by 199 · cited 1×
“In general the substituted modality is vision with the substituting modalities touch [11]–[14] or audition [15]–[17].”
BackgroundNavigation with a sensory substitution device in congenitally blind individuals
Chebat, D.-R., Schneider, F. C., Kupers, R., & Ptito, M. (2011) · Neuroreport · doi:10.1097/wnr.0b013e3283462def · Cited by 132
‘Visual’ Acuity of the Congenitally Blind Using Visual-to-Auditory Sensory Substitution
Striem-Amit, E., Guendelman, M., & Amedi, A. (2012) · PLoS ONE · doi:10.1371/journal.pone.0033136 · Cited by 117 · cited 1×
“dynamic range of the auditory nerve to provide a higher theoretical and functional acuity [21], [22].”
BackgroundTactile-‘visual’ acuity of the tongue in early blind individuals
Chebat, D.-R., Rainville, C., Kupers, R., & Ptito, M. (2007) · Neuroreport · doi:10.1097/wnr.0b013e3282f2a63 · Cited by 88 · cited 1×
“is adequate for tasks such as simple shape recognition, localization and navigation [18]–[20].”
BackgroundProulx, M. J., Stoerig, P., Ludowig, E., & Knoll, I. (2008) · PLoS ONE · doi:10.1371/journal.pone.0001840 · Cited by 81
How well do you see what you hear? The acuity of visual-to-auditory sensory substitution
Haigh, A., Brown, D. J., Meijer, P., & Proulx, M. J. (2013) · Frontiers in Psychology · doi:10.3389/fpsyg.2013.00330 · Cited by 72
Generalized Learning of Visual-to-Auditory Substitution in Sighted Individuals
Kim, J.-K., & Zatorre, R. J. (2008) · Brain Research · doi:10.1016/j.brainres.2008.06.038 · Cited by 62
Pattern recognition using a device substituting audition for vision in blindfolded sighted subjects
Poirier, C., De Volder, A., Tranduy, D., & Scheiber, C. (2007) · Neuropsychologia · doi:10.1016/j.neuropsychologia.2006.09.018 · Cited by 46
Visual objects in the auditory system in sensory substitution: How much information do we need?
Brown, D. J., Simpson, A. J. R., & Proulx, M. J. (2014) · Multisensory Research · doi:10.1163/22134808-00002462 · Cited by 23
Self-citation
Cited by
SoundSight: a mobile sensory substitution device that sonifies colour, distance, and temperature
Hamilton-Fletcher, G., Alvarez, J., Obrist, M., & Ward, J. (2022) · Journal on Multimodal User Interfaces · doi:10.1007/s12193-021-00376-w · Cited by 28
“In this approach, basic shapes and textures remain intuitive for users [14, 16], but users can have difficulty in counting and tracking multiple simultaneous frequencies in a soundscape [14, 30, 31].”
BackgroundAudio description from image by modal translation network
Ning, H., Zheng, X., Yuan, Y., & Lu, X. (2021) · Neurocomputing · doi:10.1016/j.neucom.2020.10.053 · Cited by 23
Guidelines for quantitative and qualitative studies of sensory substitution experience
Kałwak, W., Reuter, M., Łukowska, M., Majchrowicz, B., & Wierzchoń, M. (2018) · Adaptive Behavior · doi:10.1177/1059712318771690 · Cited by 14
Autoencoding sensory substitution
Tóth, V., & Parkkonen, L. (2019) · Unpublished · doi:10.13140/rg.2.2.10576.87048 · Cited by 7
“There is a trade-off between the length of the soundscape and the loss of information the sound is to convey [11]: the longer the soundscape is, the more detailed is the visual information one can extract.”
BackgroundMethod
Metadata
- Authors
- Brown, D. J., & Proulx, M. J.
- Venue
- IEEE Journal of Selected Topics in Signal Processing
- Published
- 2016-08
- Volume
- 10
- Issue
- 5
- Pages
- 924-931
- ISSN
- 1932-4553 (ISSN-L)
- 1941-0484
- OpenAlex
- W2326114782
- Cited by
- 31
- References
- 20
- FWCI
- 1.76
- Type
- journal article
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.