Transforming 3D Coloured Pixels into Musical Instrument Notes for Vision Substitution Applications
Open accessThe DOI matches its Crossref record; 3 of the 3 references it cites were found in a registry or an index.
Abstract
The goal of the See ColOr project is to achieve a noninvasive mobility aid for blind users that will use the auditory pathway to represent in real-time frontal image scenes. We present and discuss here two image processing methods that were experimented in this work: image simplification by means of segmentation, and guiding the focus of attention through the computation of visual saliency. A mean shift segmentation technique gave the best results, but for real-time constraints we simply implemented an image quantification method based on the HSL colour system. More particularly, we have developed two prototypes which transform HSL coloured pixels into spatialised classical instrument sounds lasting for 300 ms. Hue is sonified by the timbre of a musical instrument, saturation is one of four possible notes, and luminosity is represented by bass when luminosity is rather dark and singing voice when it is relatively bright. The first prototype is devoted to static images on the computer screen, while the second has been built up on a stereoscopic camera which estimates depth by triangulation. In the audio encoding, distance to objects was quantified into four duration levels. Six participants with their eyes covered by a dark tissue were trained to associate colours with musical instruments and then asked to determine on several pictures, objects with specific shapes and colours. In order to simplify the protocol of experiments, we used a tactile tablet, which took the place of the camera. Overall, colour was helpful for the interpretation of image scenes. Moreover, preliminary results with the second prototype consisting in the recognition of coloured balloons were very encouraging. Image processing techniques such as saliency could accelerate in the future the interpretation of sonified image scenes.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
SURF: Speeded Up Robust Features
Bay, H., Tuytelaars, T., & Van Gool, L. (2006) · Lecture Notes in Computer Science · doi:10.1007/11744023_32 · Cited by 14,955
An experimental system for auditory image representations
Meijer, P. B. L. (1992) · IEEE Transactions on Biomedical Engineering · doi:10.1109/10.121642 · Cited by 900
Capelle, C., Trullemans, C., Arno, P., & Veraart, C. (1998) · IEEE Transactions on Biomedical Engineering · doi:10.1109/10.720206 · Cited by 210
Research works printed but not linked: 13
- 10.1007/BF00309042 · doi:10.1007/bf00309042
- 10.1068/p220363 · DOI
- 1974, The Radio and Electronic Engineer, 44, 605, 10.1049/ree.1974.0148 · DOI · Paper page
- 10.1098/rspb.1999.0942 · DOI · Paper page
- 1965, Biometrics, 21, 768
- 1990, Introduction to Statistical Pattern Recognition
- 10.1109/34.400568 · DOI · Paper page
- 2004, Corela, 2
- 10.1016/S0010-0277(96)00791-3 · DOI · Paper page
- 10.1109/34.730558 · DOI · Paper page
- 10.1023/A:1012460413855 · DOI · Paper page
- 1977, Journal of the Audio Engineering Society, 25, 1064
- 10.2307/3680991 · DOI · Paper page
Cited by
A Systematic Review of Mapping Strategies for the Sonification of Physical Quantities
Dubus, G., & Bresin, R. (2013) · PLoS ONE · doi:10.1371/journal.pone.0082491 · Cited by 272
Abboud, S., Hanassy, S., Levy-Tzedek, S., Maidenbaum, S., & Amedi, A. (2014) · Restorative Neurology and Neuroscience · doi:10.3233/rnn-130338 · Cited by 199
“SeeColOr, for example, used timbre information to represent color hues, pitch to represent saturation and an addition of two extra timbres, simultaneous to the hue timbres, to represent luminance.”
Navigating from a Depth Image Converted into Sound
Stoll, C., Palluel-Germain, R., Fristot, V., Pellerin, D., Alleysson, D., & Graff, C. (2015) · Applied Bionics and Biomechanics · doi:10.1155/2015/543492 · Cited by 45
Deep Learning-Based Optimization of Visual–Auditory Sensory Substitution
Kim, M., Park, Y., Moon, K., & Jeong, C. Y. (2023) · IEEE Access · doi:10.1109/access.2023.3243641 · Cited by 12
“EyeMusic [17] allows the user to distinguish between five colors, whereas See ColOr [18] delivers seven colors.”
BackgroundMethodAutoencoding sensory substitution
Tóth, V., & Parkkonen, L. (2019) · Unpublished · doi:10.13140/rg.2.2.10576.87048 · Cited by 7
“EyeMusic [151] combines See ColOr and TV, performing color clustering, synthesizing soundscapes column-by-column, and ceiling the frequency at 1568 Hz, assuring the pleasantness of the auditory stimulus; albeit, the frequency cut-off reduces the auditory coding space by half.”
BackgroundMethodInfluential
Metadata
- Authors
- Bologna, G., Deville, B., Pun, T., & Vinckenbosch, M.
- Venue
- EURASIP Journal on Image and Video Processing
- Published
- 2007
- Volume
- 2007
- Pages
- 1-14
- ISSN
- 1687-5176 (ISSN-L)
- OpenAlex
- W2069430629
- Cited by
- 42
- References
- 3
- FWCI
- 2.78
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.