Visual objects in the auditory system in sensory substitution: How much information do we need?
Open accessAbstract
Sensory substitution devices such as The vOICe convert visual imagery into auditory soundscapes and can provide a basic ‘visual’ percept to those with visual impairment. However, it is not known whether technical or perceptual limits dominate the practical efficacy of such systems. By manipulating the resolution of sonified images and asking naïve sighted participants to identify visual objects through a six-alternative forced-choice procedure (6AFC) we demonstrate a ‘ceiling effect’ at 8 × 8 pixels, in both visual and tactile conditions, that is well below the theoretical limits of the technology. We discuss our results in the context of auditory neural limits on the representation of ‘auditory’ objects in a cortical hierarchy and how perceptual training may be used to circumvent these limitations.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
There is an upper limit to representation of auditory objects due to intrinsic limits of transmission in auditory cortical areas.
“Brown, Simpson and Proulx (2014) found that there is an upper limit to representation of auditory objects due to intrinsic limits of transmission in auditory cortical areas.”
From Designing sensory-substitution devices: Principles, pitfalls and potential1 (2016) · cites this paper as Brown, Simpson and Proulx (2014)
The Discussion reports an apparent 8 × 8-pixel ceiling in object recognition by naïve sensory-substitution users and proposes that limits in the ascending auditory hierarchy may explain it. It does not establish intrinsic cortical transmission limits as the cause; the authors describe that explanation as a postulate. Page numbers are not provided in our copy. (LLM, conf=0.88)
Experimental — accuracy not yet measuredBasic processing properties of the perceptual channels in question need to be considered and effective models of encoding the information into sound and haptics must be developed.
“The basic processing properties of the perceptual channels in question need to be considered and effective models of encoding the information into sound and haptics must be developed (see e.g. Brown et al., 2014)”
From Designing sensory-substitution devices: Principles, pitfalls and potential1 (2016) · cites this paper as Brown et al., 2014
The Introduction (page not available in our copy) says that sensory-substitution design must account for auditory capacity and describes how visual information is encoded in sound. It also notes that touch-receptor distribution constrains tactile-device resolution, but does not call for developing effective models for encoding information into haptics. (LLM, conf=0.84)
Experimental — accuracy not yet measuredBrown et al. (2014) make the case that there may be upper resolution limits for sonification of visual information.
“Brown et al. (2014) make the case that there may be upper resolution limits for sonification of visual information.”
From Designing sensory-substitution devices: Principles, pitfalls and potential1 (2016) · cites this paper as Brown et al. (2014)
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
The sensations produced by electrical stimulation of the visual cortex
Brindley, G. S., & Lewin, W. S. (1968) · The Journal of Physiology · doi:10.1113/jphysiol.1968.sp008519 · Cited by 1,170 · see citing works
An experimental system for auditory image representations
Meijer, P. B. L. (1992) · IEEE Transactions on Biomedical Engineering · doi:10.1109/10.121642 · Cited by 900 · see citing works
Schmidt, E. M., Bak, M. J., Hambrecht, F. T., Kufta, C. V., O'Rourke, D. K., & Vallabhanath, P. (1996) · Brain · doi:10.1093/brain/119.2.507 · Cited by 672 · see citing works
Dobelle, W. H., & Mladejovsky, M. G. (1974) · The Journal of Physiology · doi:10.1113/jphysiol.1974.sp010766 · Cited by 468 · see citing works
Shape conveyed by visual-to-auditory sensory substitution activates the lateral occipital complex
Amedi, A., Stern, W. M., Camprodon, J. A., Bermpohl, F., Merabet, L., Rotman, S., Hemond, C., Meijer, P., & Pascual-Leone, A. (2007) · Nature Neuroscience · doi:10.1038/nn1912 · Cited by 451 · see citing works
Dobelle, W. H., Mladejovsky, M. G., & Girvin, J. P. (1974) · Science · doi:10.1126/science.183.4123.440 · Cited by 440 · see citing works
Pattern recognition with the optic nerve visual prosthesis
Veraart, C., Wanet‐Defalque, M., Gérard, B., Vanlierde, A., & Delbeke, J. (2003) · Artificial Organs · doi:10.1046/j.1525-1594.2003.07305.x · Cited by 259 · see citing works
Brain Plasticity: ‘Visual’ Acuity of Blind Persons via the Tongue
Sampaio, E., Maris, S., & Bach-y-Rita, P. (2001) · Brain Research · doi:10.1016/s0006-8993(01)02667-1 · Cited by 258 · see citing works
Abboud, S., Hanassy, S., Levy-Tzedek, S., Maidenbaum, S., & Amedi, A. (2014) · Restorative Neurology and Neuroscience · doi:10.3233/rnn-130338 · Cited by 199 · see citing works
Arno, P., De Volder, A. G., Vanlierde, A., Wanet-Defalque, M.-C., Streel, E., Robert, A., Sanabria-Bohórquez, S., & Veraart, C. (2001) · NeuroImage · doi:10.1006/nimg.2000.0731 · Cited by 148 · see citing works
Tactile-‘visual’ acuity of the tongue in early blind individuals
Chebat, D.-R., Rainville, C., Kupers, R., & Ptito, M. (2007) · Neuroreport · doi:10.1097/wnr.0b013e3282f2a63 · Cited by 88 · see citing works
Proulx, M. J., Stoerig, P., Ludowig, E., & Knoll, I. (2008) · PLoS ONE · doi:10.1371/journal.pone.0001840 · Cited by 81 · see citing works
How well do you see what you hear? The acuity of visual-to-auditory sensory substitution
Haigh, A., Brown, D. J., Meijer, P., & Proulx, M. J. (2013) · Frontiers in Psychology · doi:10.3389/fpsyg.2013.00330 · Cited by 72 · see citing works
Spatial updating depends on gaze direction even after loss of vision
Reuschel, J., Rösler, F., Henriques, D. Y. P., & Fiehler, K. (2012) · The Journal of Neuroscience · doi:10.1523/jneurosci.2714-11.2012 · Cited by 25 · see citing works
Flagged first, then by citations
Research works printed but not linked: 18
- Ahissar, M., Nahum, M., Nelken, I. and Hochstein, S. (2009). Reverse hierarchies and sensory learning, Phil. Trans. R. Soc. B 364, 285-299.
- Bach-y-Rita, P., Collins, C. C., Saunders, F. A., White, B. and Scadden, L. (1969). Vision sub- stitution by tactile image projection, Nature 221(5184), 963-964.
- Barton, B., Venezia, J. H., Saberi, K., Hickok, G. and Brewer, A. A. (2012). Orthogonal acoustic dimensions define auditory field maps in human cortex, Proc. Natl Acad. Sci. USA 109, 20738-20743.
- Benav, H., Bartz-Schmidt, K. U., Besch, D., Bruckmann, A., Gekeler, F., Greppmaier, U., Harscher, A., Kibbel, S., Kusnyerik, A., Peters, T., Sachs, H., Stett, A., Stingl, K., Wil- helm, B., Wilke, R., Wrobel, W. and Zrenner, E. (2010). Restoration of useful vision up to letter recognition capabilities using subretinal microphotodiodes, Conf. Proc. IEEE Eng. Med. Biol. Soc. 2010, 5919-5922.
- Brown, D. J. and Proulx, M. J. (2013). Increased signal complexity improves the breadth of generalization in auditory perceptual learning, Neural Plast. 879047. DOI:10.1155/2013/ 879047. · DOI
- Capelle, C., Trullemans, C., Arno, P. and Veraart, C. (1998). A real-time experimental prototype for enhancement of vision rehabilitation using auditory substitution, IEEE Trans. Biomed. Eng. 45, 1279-1293.
- Chai, X., Li, L., Wu, K., Zhou, C., Cao, P. and Ren, Q. (2008a). C-sight visual prostheses for the blind, IEEE Eng. Med. Biol. Mag. 27(5), 20-28.
- Ding, N. and Simon, J. Z. (2012). Emergence of neural encoding of auditory objects while listening to competing speakers, Proc. Natl Acad. Sci. USA 109, 11854-11859.
- Keseru, M., Feucht, M., Bornfeld, N., Laube, T., Walter, P., Rossler, G., Velikay-Parel, M., Hornig, R. and Richard, G. (2012). Acute electrical stimulation of the human retina with an epiretinal electrode array, Acta Ophthalmol. 90, e1-e8. DOI:10.1111/j.1755-3768.2011. 02288.x. · DOI
- Mesgarani, N. and Chang, E. F. (2012). Selective cortical representation of attended speaker in multi-talker speech perception, Nature 485(7397), 233-236.
- Pasley, B. N., David, S. V., Mesgarani, N., Flinker, A., Shamma, S. A., Crone, N. E., Knight, R. T. and Chang, E. F. (2012). Reconstructing speech from human auditory cortex, PLoS Biol. 10(1), e1001251. DOI:10.1371/journal.pbio.1001251. · DOI · Paper page
- Plaza, P., Cuevas, I., Grandin, C., De Volder, A. G. and Renier, L. (2012). Looking into task- specific activation using a prosthesis substituting vision with audition, ISRN Rehabil. 2012, 15. DOI:10.5402/2012/490950. · DOI
- Simon, J. Z. and Ding, N. (2010). Magnetoencephalography and auditory neural representa- tions, 26th Southern Biomedical Engineering Conference: Sbec 2010, 32, 45-48.
- Simpson, A. J. and Reiss, J. D. (2013). The dynamic range paradox: a central auditory model of intensity change detection, PLoS One 8(2), e57497. DOI:10.1371/journal.pone.0057497. · DOI
- Simpson, A. J., Reiss, J. D. and McAlpine, D. (2013). Tuning of human modulation filters is carrier-frequency dependent, PLoS One 8(8), e73590. DOI:10.1371/journal.pone.0073590. · DOI
- Striem-Amit, E., Guendelman, M. and Amedi, A. (2012). 'Visual' acuity of the con- genitally blind using visual-to-auditory sensory substitution, Plos One 7(3), e33136. DOI:10.1371/journal.pone.0033136. · DOI · Paper page
- Teki, S., Chait, M., Kumar, S., Shamma, S. and Griffiths, T. D. (2013). Segregation of complex acoustic scenes based on temporal coherence, Elife 2, e00699. DOI:10.7554/eLife.00699. · DOI
- Zrenner, E., Bartz-Schmidt, K. U., Benav, H., Besch, D., Bruckmann, A., Gabel, V.-P., Gekeler, F., Greppmaier, U., Harscher, A., Kibbel, S., Koch, J., Kusnyerik, A., Peters, T., Stingl, K., Sachs, H., Stett, A., Szurman, P., Wilhelm, B. and Wilke, R. (2011). Subretinal elec- tronic chips allow blind patients to read letters and combine them to words, Proc. Biol. Sci. 278(1711), 1489-1497.
Cited by
Sorted by citations
Designing sensory-substitution devices: Principles, pitfalls and potential1
Kristjánsson, Á., Moldoveanu, A., Jóhannesson, Ó. I., Balan, O., Spagnol, S., Valgeirsdóttir, V. V., & Unnthorsson, R. (2016) · Restorative Neurology and Neuroscience · doi:10.3233/rnn-160647 · Cited by 116 · see citing works
“Brown et al. (2014) make the case that there may be upper resolution limits for sonification of visual information.”
BackgroundMethodInfluentialProulx, M. J., Gwinnutt, J., Dell’Erba, S., Levy-Tzedek, S., de Sousa, A. A., & Brown, D. J. (2015) · Restorative Neurology and Neuroscience · doi:10.3233/rnn-150541 · Cited by 36 · see citing works
Self-citationSoundSight: a mobile sensory substitution device that sonifies colour, distance, and temperature
Hamilton-Fletcher, G., Alvarez, J., Obrist, M., & Ward, J. (2022) · Journal on Multimodal User Interfaces · doi:10.1007/s12193-021-00376-w · Cited by 28 · see citing works
- Authors
- Brown, D. J., Simpson, A. J. R., & Proulx, M. J.
- Affiliations
- Queen Mary University of London (GB)
- University of Bath (GB)
- ORCID
- Proulx, Michael J.
- Venue
- Multisensory Research
- Publisher
- Walter de Gruyter GmbH
- Published
- 2014
- Volume
- 27
- Issue
- 5-6
- Pages
- 337-357
- ISSN
- 2213-4794 (ISSN-L)
- 2213-4808
- OpenAlex
- W2154733620
- Cited by
- 23
- References
- 14
- Citations by year
- 2026: 1 · 2025: 1 · 2023: 1 · 2022: 3 · 2021: 2 · 2020: 2 · 2019: 1 · 2017: 1 · 2016: 5 · 2015: 5 · 2014: 1
- FWCI
- 2.23
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Locations
- Multisensory Research (publishedVersion, cc-by)
- PubMed (publishedVersion)
- The University of Bath Online Publications Store (The University of Bath) (submittedVersion, cc-by)
- Pure (University of Bath) (submittedVersion, cc-by)
- Funders
- Queen Mary University of London
- Engineering and Physical Sciences Research Council (EP/J017205/1)
- Topics
- Life Sciences › Neuroscience › Cognitive Neuroscience › Tactile and Sensory Interactions
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Multisensory perception and integration
- Life Sciences › Neuroscience › Cognitive Neuroscience › Visual perception and processing mechanisms
- Keywords
- sensory substitution, voice, sonification, visual impairment, image resolution, ceiling effect, perceptual training, visual imagery, tactile perception
- MeSH terms
- Adolescent, Adult, Auditory Perception, Blindness, Discrimination, Psychological, Female, Form Perception, Humans, Male, Self-Help Devices, Sensory Deprivation, Touch Perception, Young Adult
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
