Colorophone 2.0: A Wearable Color Sonification Device Generating Live Stereo-Soundscapes—Design, Implementation, and Usability Audit
Open accessThe DOI matches its Crossref record; 72 of the 73 references it cites were found in a registry or an index.
Abstract
The successful development of a system realizing color sonification would enable auditory representation of the visual environment. The primary beneficiary of such a system would be people that cannot directly access visual information—the visually impaired community. Despite the plethora of sensory substitution devices, developing systems that provide intuitive color sonification remains a challenge. This paper presents design considerations, development, and the usability audit of a sensory substitution device that converts spatial color information into soundscapes. The implemented wearable system uses a dedicated color space and continuously generates natural, spatialized sounds based on the information acquired from a camera. We developed two head-mounted prototype devices and two graphical user interface (GUI) versions. The first GUI is dedicated to researchers, and the second has been designed to be easily accessible for visually impaired persons. Finally, we ran fundamental usability tests to evaluate the new spatial color sonification algorithm and to compare the two prototypes. Furthermore, we propose recommendations for the development of the next iteration of the system.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures














Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Stevens, S. S. (1957) · Psychological Review · doi:10.1037/h0046162 · Cited by 3,361
Auditory Scene Analysis: The Perceptual Organization of Sound
Bregman, A. S., & McAdams, S. (1994) · The Journal of the Acoustical Society of America · doi:10.1121/1.408434 · Cited by 3,029
Crossmodal correspondences: A tutorial review
Spence, C. (2011) · Attention, Perception, & Psychophysics · doi:10.3758/s13414-010-0073-7 · Cited by 1,618 · cited 1×
“When selecting the number of color components to be represented by a sonification algorithm, we should bear in mind that if this number is too large, it will be difficult for a naïve user to remember and recognize all the color–sounds associations.”
MethodA Scale for the Measurement of the Psychological Magnitude Pitch
Stevens, S. S., Volkmann, J., & Newman, E. B. (1937) · The Journal of the Acoustical Society of America · doi:10.1121/1.1915893 · Cited by 1,107 · cited 2×
“Be pleasant for the user [25]; in amplitude corresponding to the maximal color intensity to avoid masking chosen cross-modal correspondences yellow—middle-high pitch,”
BackgroundMethodMills, A. W. (1958) · The Journal of the Acoustical Society of America · doi:10.1121/1.1909553 · Cited by 964
An experimental system for auditory image representations
Meijer, P. B. L. (1992) · IEEE Transactions on Biomedical Engineering · doi:10.1109/10.121642 · Cited by 900
Wearable Obstacle Avoidance Electronic Travel Aids for Blind: A Survey
Dakopoulos, D., & Bourbakis, N. G. (2010) · IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews) · doi:10.1109/tsmcc.2009.2021255 · Cited by 654
Fastl, H., & Zwicker, E. (2007) · Springer eBooks · doi:10.1007/978-3-540-68888-4 · Cited by 537
Foster, D. H. (2011) · Vision Research · doi:10.1016/j.visres.2010.09.006 · Cited by 505
Ebner, M. (2007) · Cited by 482
What is the Bandwidth of Perceptual Experience?
Cohen, M. A., Dennett, D. C., & Kanwisher, N. (2016) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2016.03.006 · Cited by 419
Listener weighting of cues for lateral angle: The duplex theory of sound localization revisited
Macpherson, E. A., & Middlebrooks, J. C. (2002) · The Journal of the Acoustical Society of America · doi:10.1121/1.1471898 · Cited by 414
Early- and Late-Onset Blind Individuals Show Supra-Normal Auditory Abilities in Far-Space
Voss, P., Lassonde, M., Gougoux, F., Fortin, M., Guillemot, J.-P., & Lepore, F. (2004) · Current Biology · doi:10.1016/j.cub.2004.09.051 · Cited by 308
Bhowmick, A., & Hazarika, S. M. (2017) · Journal on Multimodal User Interfaces · doi:10.1007/s12193-016-0235-6 · Cited by 271
Histomorphometry of the human optic nerve.
Jonas, J. B., Müller-Bergh, J. A., Schlötzer-Schrehardt, U. M., & Naumann, G. O. H. (1990) · PubMed · Cited by 263
Impairment of auditory spatial localization in congenitally blind human subjects
Gori, M., Sandini, G., Martinoli, C., & Burr, D. C. (2014) · Brain · doi:10.1093/brain/awt311 · Cited by 252
Analysis of the human auditory nerve
Spoendlin, H., & Schrott, A. (1989) · Hearing Research · doi:10.1016/0378-5955(89)90056-7 · Cited by 247
Maidenbaum, S., Abboud, S., & Amedi, A. (2014) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2013.11.007 · Cited by 230
Properties of auditory stream formation
Moore, B. C. J., & Gockel, H. E. (2012) · Philosophical Transactions of the Royal Society B: Biological Sciences · doi:10.1098/rstb.2011.0355 · Cited by 217
Wearable assistive devices for visually impaired: A state of the art survey
Tapu, R., Mocanu, B., & Zaharia, T. (2020) · Pattern Recognition Letters · doi:10.1016/j.patrec.2018.10.031 · Cited by 211
Abboud, S., Hanassy, S., Levy-Tzedek, S., Maidenbaum, S., & Amedi, A. (2014) · Restorative Neurology and Neuroscience · doi:10.3233/rnn-130338 · Cited by 199 · cited 2×
“The existing color sonification methods applied in SSDs can be divided into three categories: the first category contains systems that directly associate light frequency with sound frequency [21,22], the second category systems use associations between a predefined color category and the presented sound [20,46] (e.”
MethodSpatial neglect—a vestibular disorder?
Karnath, H.-O., & Dieterich, M. (2006) · Brain · doi:10.1093/brain/awh698 · Cited by 192
Csapó, Á., Wersényi, G., Nagy, H., & Stockman, T. (2015) · Journal on Multimodal User Interfaces · doi:10.1007/s12193-015-0182-7 · Cited by 181 · cited 1×
“Together with observed progress in the electronic systems field [12], these encouraging results fuel the intensive growth in the development of novel SSDs in recent years [13,14].”
BackgroundAssisting the Visually Impaired: Obstacle Detection and Warning System by Acoustic Feedback
Rodríguez, A., Yebes, J. J., Alcantarilla, P., Bergasa, L., Almazán, J., & Cela, A. (2012) · Sensors · doi:10.3390/s121217476 · Cited by 181
Minimum audible angle thresholds for sources varying in both elevation and azimuth
Perrott, D. R., & Saberi, K. (1990) · The Journal of the Acoustical Society of America · doi:10.1121/1.399421 · Cited by 165
Estimation of a Spherical-Head Model from Anthropometry
Algazi, V. R., Avendaño, C., & Duda, R. O. (2001) · Journal of the Audio Engineering Society · Cited by 160
Rehabilitation of spatial neglect by prism adaptation
Jacquin‐Courtois, S., O’Shea, J., Luauté, J., Pisella, L., Revol, P., Mizuno, K., Rode, G., & Rossetti, Y. (2013) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2013.02.007 · Cited by 155
Ricciardi, E., Bonino, D., Pellegrini, S., & Pietrini, P. (2014) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2013.10.006 · Cited by 154
Chandler, D. W., & Grantham, D. W. (1992) · The Journal of the Acoustical Society of America · doi:10.1121/1.402443 · Cited by 145
How many pixels make an image?
TORRALBA, A. (2009) · Visual Neuroscience · doi:10.1017/s0952523808080930 · Cited by 141
Iachini, T., Ruggiero, G., & Ruotolo, F. (2014) · Behavioural Brain Research · doi:10.1016/j.bbr.2014.07.032 · Cited by 119
Designing sensory-substitution devices: Principles, pitfalls and potential1
Kristjánsson, Á., Moldoveanu, A., Jóhannesson, Ó. I., Balan, O., Spagnol, S., Valgeirsdóttir, V. V., & Unnthorsson, R. (2016) · Restorative Neurology and Neuroscience · doi:10.3233/rnn-160647 · Cited by 116 · cited 5×
“…and functionality None-theless, the first prototype had multiple issues regarding its usability and functionality. the previous implementation, we used a standard webcam attached to a appeared to be bulky Firstly, in the previous implementation, we used a standard webcam attached to a head-band,…”
BackgroundMethodInfluentialMapping unpleasantness of sounds to their auditory representation
Kumar, S., Forster, H. M., Bailey, P., & Griffiths, T. D. (2008) · The Journal of the Acoustical Society of America · doi:10.1121/1.3006380 · Cited by 94
Is there a future for sensory substitution outside academic laboratories?
Elli, G. V., Benetti, S., & Collignon, O. (2014) · Multisensory Research · doi:10.1163/22134808-00002460 · Cited by 90
Vercillo, T., Burr, D. C., & Gori, M. (2016) · Developmental Psychology · doi:10.1037/dev0000103 · Cited by 86
Reich, L., Maidenbaum, S., & Amedi, A. (2012) · Current Opinion in Neurology · doi:10.1097/wco.0b013e32834ed723 · Cited by 78
Spatial audio through a bone conduction interface
MacDonald, J. A., Henry, P. P., & Letowski, T. R. (2006) · International Journal of Audiology · doi:10.1080/14992020600876519 · Cited by 77
Minimum audible movement angle as a function of signal frequency and the velocity of the source
Perrott, D. R., & Tucker, J. (1988) · The Journal of the Acoustical Society of America · doi:10.1121/1.395908 · Cited by 77
Auditory augmented reality: Object sonification for the visually impaired
Ribeiro, F., Florencio, D., Chou, P. A., & Zhang, Z. (2012) · 2012 IEEE 14th International Workshop on Multimedia Signal Processing (MMSP) · doi:10.1109/mmsp.2012.6343462 · Cited by 75
Auditory Spatial Perception: Auditory Localization
Letowski, T. R., & Letowski, S. (2012) · Defense Technical Information Center (DTIC) · doi:10.21236/ada562292 · Cited by 73
Electronic interfaces aiding the visually impaired in environmental access, mobility and navigation
Strumillo, P. (2010) · 3rd International Conference on Human System Interaction · doi:10.1109/hsi.2010.5514595 · Cited by 71
The supramodal brain: implications for auditory perception
Rosenblum, L. D., Dias, J. W., & Dorsi, J. (2016) · Journal of Cognitive Psychology · doi:10.1080/20445911.2016.1181691 · Cited by 66
Spatial Competence and Brain Plasticity in Congenital Blindness via Sensory Substitution Devices
Chebat, D., Schneider, F., & Ptito, M. (2020) · Frontiers in Neuroscience · doi:10.3389/fnins.2020.00815 · Cited by 61
Hearing colors: an example of brain plasticity
Alfaro, A., Bernabéu, Á., Mateo, C. M., Parra, J., & Fernández, E. (2015) · Frontiers in Systems Neuroscience · doi:10.3389/fnsys.2015.00056 · Cited by 58
Minimum Audible Movement Angle as a Function of the Azimuth and Elevation of the Source
Strybel, T. Z., Manllgas, C. L., & Perrott, D. R. (1992) · Human Factors: The Journal of the Human Factors and Ergonomics Society · doi:10.1177/001872089203400302 · Cited by 56
The Causes and Consequences of Color Vision
Gerl, E. J., & Morris, M. R. (2008) · Evolution Education and Outreach · doi:10.1007/s12052-008-0088-x · Cited by 50
A Stereo Image Processing System For Visually Impaired
Balakrishnan, G., Sainarayanan, G., Nagarajan, R., & Yaacob, S. (2008) · Zenodo (CERN European Organization for Nuclear Research) · doi:10.5281/zenodo.1075125 · Cited by 50
The optophone: an electronic blind aid
Capp, M., & Picton, P. (2000) · Engineering Science & Education Journal · doi:10.1049/esej:20000306 · Cited by 49
Aerial Obstacle Detection With 3-D Mobile Devices
Saez, J. M., Escolano, F., & Lozano, M. Á. (2014) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2014.2322392 · Cited by 47
Multisensory integration, sensory substitution and visual rehabilitation
Proulx, M. J., Ptito, M., & Amedi, A. (2014) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2014.03.004 · Cited by 45
Jóhannesson, Ó. I., Balan, O. C., Unnþórsson, R., Moldoveanu, A. D. B., & Kristjansson, A. K. (2016) · Brain Sciences · doi:10.3390/brainsci6030020 · Cited by 45
Stereo Vision Based Sensory Substitution for the Visually Impaired
Caraiman, S., Zvoristeanu, O., Burlacu, A., & Herghelegiu, P. (2019) · Sensors · doi:10.3390/s19122771 · Cited by 44
Adaptation to prism-displaced vision: The importance of target-pointing
Welch, R. B. (1969) · Perception & Psychophysics · doi:10.3758/bf03209569 · Cited by 42
SOUNDVIEW: SENSING COLOR IMAGES BY KINESTHETIC AUDIO
van den Doel, K. (2003) · SMARTech Repository (Georgia Institute of Technology) · Cited by 39
On the use of the auditory pathway to represent image scenes in real-time
Bologna, G., deVille, B., & Pun, T. (2008) · Neurocomputing · doi:10.1016/j.neucom.2008.06.020 · Cited by 39
Color Sonification for the Visually Impaired
Cavaco, S., Henriques, J. T., Mengucci, M., Correia, N., & Medeiros, F. (2013) · Procedia Technology · doi:10.1016/j.protcy.2013.12.117 · Cited by 39
Cross-Modal Correspondences Enhance Performance on a Colour-to-Sound Sensory Substitution Device
Hamilton-Fletcher, G., Wright, T. D., & Ward, J. (2016) · Multisensory Research · doi:10.1163/22134808-00002519 · Cited by 38
Are red, yellow, green, and blue perceptual categories?
Witzel, C., & Gegenfurtner, K. R. (2018) · Vision Research · doi:10.1016/j.visres.2018.04.002 · Cited by 36
Sensorimotor theory of consciousness
Degenaar, J., & O'Regan, J. K. (2015) · Scholarpedia · doi:10.4249/scholarpedia.4952 · Cited by 35
Sonification: Review of Auditory Display Solutions in Electronic Travel Aids for the Blind
Bujacz, M., & Strumiłło, P. (2016) · Archives of Acoustics · doi:10.1515/aoa-2016-0040 · Cited by 33
Representing Colour Through Hearing and Touch in Sensory Substitution Devices
Hamilton-Fletcher, G., & Ward, J. (2013) · Multisensory Research · doi:10.1163/22134808-00002434 · Cited by 32
Kuehni, R. G. (2001) · Color Research & Application · doi:10.1002/1520-6378(200104)26:2<171::aid-col1008>3.0.co;2-4 · Cited by 32
Comparison of auditory spatial bisection and minimum audible angle in front, lateral, and back space
Aggius-Vella, E., Kolarik, A. J., Gori, M., Cirstea, S., Campus, C., Moore, B. C. J., & Pardhan, S. (2020) · Scientific Reports · doi:10.1038/s41598-020-62983-z · Cited by 29
Marshall, N. J., Baraas, R. C., & Kremers, J. (2016) · Springer eBooks · doi:10.1007/978-3-319-44978-4 · Cited by 21
Musical Vision: an interactive bio-inspired sonification tool to convert images into music
Polo, A., & Sevillano, X. (2018) · Journal on Multimodal User Interfaces · doi:10.1007/s12193-018-0280-4 · Cited by 21
Brown, D. J., Simpson, A. J. R., & Proulx, M. J. (2015) · Frontiers in Psychology · doi:10.3389/fpsyg.2015.01522 · Cited by 18
HUE MUSIC - CREATING TIMBRAL SOUNDSCAPES FROM COLOURED PICTURES
Payling, D., Mills, S., & Howle, T. (2007) · SMARTech Repository (Georgia Institute of Technology) · Cited by 15
A Sensory Substitution Device Inspired by the Human Visual System
Osinski, D., & Hjelme, D. R. (2018) · 2018 11th International Conference on Human System Interaction (HSI) · doi:10.1109/hsi.2018.8431078 · Cited by 15 · cited 8×
“One of these was the first version of the Colorophone system [25].”
BackgroundMethodInfluentialSelf-citationVisual Echolocation Concept for the Colorophone Sensory Substitution Device Using Virtual Reality
Bizoń-Angov, P., Osiński, D., Wierzchoń, M., & Konieczny, J. (2021) · Sensors · doi:10.3390/s21010237 · Cited by 15
Self-citationSonification of 3D Scenes in an Electronic Travel Aid for the Blind
Bujacz, M., Pec, M., Skulimowski, P., Strumiłło, P., & Materka, A. (2011) · InTech eBooks · doi:10.5772/15311 · Cited by 12
The opposite of green is purple?
Moroney, N. M. (2009) · Proceedings of SPIE, the International Society for Optical Engineering/Proceedings of SPIE · doi:10.1117/12.817013 · Cited by 2
Designing Auditory Color Space for Color Sonification Systems
Osiński, D., Bizoń-Angov, P., Midtfjord, H., Wierzchoń, M., & Hjelme, D. R. (2019) · International Conference on Auditory Display (ICAD) · doi:10.21785/icad2019.076 · Cited by 2
Self-citation
Research works printed but not linked: 4
- Starkiewicz, W., and Kuliszewski, T. (1963, January 1). The 80-Channel Elektroftalm. Proceedings of the International Congress on Technology and Blindness, New York, NY, USA.
- Capalbo, Z., and Glenney, B. (2009, January 1). Hearing Color: Radical Pluralistic Realism and SSDs. Proceedings of the AP-CAP, Tokyo, Japan.
- Fairchild, M.D. (2005). Color Appearance Models. Wiley-IS&T Series in Imaging Science and Technology, Wiley. [2nd ed.].
- Bologna, G., Deville, B., and Pun, T. (2010, January 9–15). Sonification of Color and Depth in a Mobility Aid for Blind People. Proceedings of the Proceedings of the 16th International Conference on Auditory Display (ICAD2010), Washington, DC, USA.
Cited by
TactualPlot: Spatializing Data as Sound Using Sensory Substitution for Touchscreen Accessibility
Chundury, P., Reyazuddin, Y., Jordan, J. B., Lazar, J. K., & Elmqvist, N. (2023) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2023.3326937 · Cited by 34
Deep Learning-Based Optimization of Visual–Auditory Sensory Substitution
Kim, M., Park, Y., Moon, K., & Jeong, C. Y. (2023) · IEEE Access · doi:10.1109/access.2023.3243641 · Cited by 12
“…could enable visually impaired and partially sighted people to recognize colors or The associate editor coordinating the review of this manuscript and approving it for publication was Felix Albu . objects [1], [2], navigate and avoid obstacles [3], [4], and even identify facial expressions [5].”
BackgroundŁukowska, M., Kałwak, W., Osiński, D., Janik, J., & Wierzchoń, M. (2022) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2022.102925 · Cited by 11
Self-citationDesign and Implementation of a Real-Time Color Recognition System for the Visually Impaired
Samara, M., Deriche, M. A., Al-Sadah, J., & Osais, Y. E. (2022) · Arabian Journal for Science and Engineering · doi:10.1007/s13369-022-07506-w · Cited by 8
Butorovа, A. S., & Sergeev, A. (2025) · The European Physical Journal Special Topics · doi:10.1140/epjs/s11734-025-01596-x · Cited by 3
Metadata
- Authors
- Osiński, D., Łukowska, M., Hjelme, D. R., & Wierzchoń, M.
- Affiliations
- Norwegian University of Science and Technology (NO)
- Jagiellonian University (PL)
- Corresponding author
- Osiński, Dominik
- Venue
- Sensors
- Publisher
- MDPI AG
- Published
- 2021-11-05
- Volume
- 21
- Issue
- 21
- Pages
- 7351
- ISSN
- 1424-8220 (ISSN-L)
- PMC
- PMC8587929
- OpenAlex
- W3213597375
- Cited by
- 11
- References
- 73
- Citations by year
- 2026: 1 · 2025: 1 · 2024: 1 · 2023: 3 · 2022: 5
- FWCI
- 1.18
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- Sensors (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Homo Politicus (Academy of Humanities and Economics in Lodz) (submittedVersion, cc-by-nc-nd)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- NORA - Norwegian Open Research Archives (submittedVersion, cc-by)
- Funders
- National Science Center (2016/23/B/HS6/00275)
- Norwegian University of Science and Technology (Kompetanseløft)
- Topics
- Life Sciences › Neuroscience › Cognitive Neuroscience › Tactile and Sensory Interactions
- Health Sciences › Health Professions › Occupational Therapy › Assistive Technology in Communication and Mobility
- Social Sciences › Arts and Humanities › Music › Music Technology and Sound Studies
- Keywords
- sensory substitution, visually impaired users, wearable devices, head-mounted devices, spatial audio, color space, usability evaluation, graphical user interface design
- MeSH terms
- Wearable Electronic Devices, Algorithms, Humans, Sound, Persons with Visual Disabilities
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.