Designing sensory-substitution devices: Principles, pitfalls and potential1
Open accessThe DOI matches its Crossref record; 224 of the 229 references it cites were found in a registry or an index; 30 of the 150 sources its claims cite were checked for support.
Abstract
An exciting possibility for compensating for loss of sensory function is to augment deficient senses by conveying missing information through an intact sense. Here we present an overview of techniques that have been developed for sensory substitution (SS) for the blind, through both touch and audition, with special emphasis on the importance of training for the use of such devices, while highlighting potential pitfalls in their design. One example of a pitfall is how conveying extra information about the environment risks sensory overload. Related to this, the limits of attentional capacity make it important to focus on key information and avoid redundancies. Also, differences in processing characteristics and bandwidth between sensory systems severely constrain the information that can be conveyed. Furthermore, perception is a continuous process and does not involve a snapshot of the environment. Design of sensory substitution devices therefore requires assessment of the nature of spatiotemporal continuity for the different senses. Basic psychophysical and neuroscientific research into representations of the environment and the most effective ways of conveying information should lead to better design of sensory substitution systems. Sensory substitution devices should emphasize usability, and should not interfere with other inter- or intramodal perceptual function. Devices should be task-focused since in many cases it may be impractical to convey too many aspects of the environment. Evidence for multisensory integration in the representation of the environment suggests that researchers should not limit themselves to a single modality in their design. Finally, we recommend active training on devices, especially since it allows for externalization, where proximal sensory stimulation is attributed to a distinct exterior object.
Overview
CoChat Summary
TL;DR
This review argues that sensory substitution devices for blindness should be designed around psychophysical limits, attentional capacity, task focus, and multisensory integration, with active training to promote externalization. It surveys haptic and auditory approaches and highlights pitfalls such as sensory overload, bandwidth mismatches, and insufficient basic research.
Key findings
- Sensory bandwidth differs greatly: the optic nerve contains over 1 million fibers versus around 30,000 in the auditory nerve, and vision has four orders of magnitude greater bandwidth than haptic perception; two-point tactile thresholds range from ∼3–4 mm on the fingers to over 40 mm on the thighs and calves.
- Training and externalization are central: Bach-y-Rita et al. (1969) found object recognition after 5–15 h of training; Sampaio et al. (2001) reported tongue-SSD acuity of 20/860 on average, doubling after 9 h of training; Striem-Amit et al. (2012a) used 73 h of vOICe training, with 55% showing acuity above the blindness threshold; Auvray et al. (2007) found object identification after 15 h of vOICe training.
- Design should avoid sensory overload and task-irrelevant information; attentional limits and within-modality interference mean devices should convey critical information and not interfere with natural audition, while multisensory integration suggests combining haptics and audition may help.
- Neural evidence indicates that “visual” areas can serve supramodal functions in blind users, with activation of regions such as LOC, VWFA, and FFA during tactile or auditory tasks, and active interaction may produce more visual-cortex activity than passive processing.
Methods
- Design: narrative review/overview of sensory substitution for blindness through touch and audition, emphasizing training, design pitfalls, and potential.
- Data or sample: synthesizes prior psychophysical, neuroimaging, TMS, animal-deprivation, and SSD studies; no new empirical sample is reported.
- Main analysis: thematic discussion of principles, pitfalls, and potential, including bandwidth, attention, externalization, multisensory integration, and training.
Limitations
- The paper states that the literature lacks clear and precise training guidelines for SSDs and that no studies have yet addressed training systematically; evidence for enhanced brain capacity through training is scarce, and video-game/brain-training claims are confounded or task-specific.
- The paper notes controversy over whether blind individuals have better tactile acuity and caveats that multisensory integration can be worse with impaired senses, with visual experience possibly needed for crossmodal calibration.
- Evident from the text: as a narrative review, it does not report a systematic search or selection protocol or primary data, and many conclusions rely on cited studies with varying designs and populations.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
26 Valid · 4 Not valid · 74 Not checked · 40 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Brown, Simpson and Proulx (2014) found that there is an upper limit to representation of auditory objects due to intrinsic limits of transmission in auditory cortical areas.”
§ Designing sensory-substitution devices: Principles, pitfalls and potential1 > Key general considerations for sensory substitution
What the sources say
- Partly supportedcites Brown, Simpson and Proulx (2014) sourceExperimental — accuracy not yet measured
The Discussion reports an apparent 8 × 8-pixel ceiling in object recognition by naïve sensory-substitution users and proposes that limits in the ascending auditory hierarchy may explain it. It does not establish intrinsic cortical transmission limits as the cause; the authors describe that explanation as a postulate. Page numbers are not provided in our copy. (LLM, conf=0.88)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Such evidence suggests that the loss of one perceptual channel can be compensated for by higher resolution activity in another (Gougoux et al., 2004; Stevens & Weaver, 2009).”
§ Designing sensory-substitution devices: Principles, pitfalls and potential1 > Neural evidence regarding sensory substitution
What the sources say
- Not checked — no full text of the sourcecites Gougoux et al., 2004 source
- Partly supportedcites Stevens & Weaver, 2009 sourceExperimental — accuracy not yet measured
The paper’s discussion of early-onset blindness (page unavailable in our copy) reports superior auditory abilities, including greater temporal and spectral processing, and proposes that auditory cortical plasticity may contribute. Its own results concern altered activity during passive listening, not a measured increase in auditory resolution. The claim also generalizes from vision and hearing to any pair of perceptual channels. (LLM, conf=0.84)
hedgedstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“The literature on multimodal interactions (see e.g. Deroy, Chen & Spence, 2014; Driver & Spence, 2000; Koelewijn, Bronkhorst & Theeuwes, 2010; Kristjánsson, Thorvaldsson, & Kristjánsson, 2014; Stein & Meredith, 1993) makes clear that a holistic picture of perceptual representation cannot be constructed without understanding multisensory interactions.”
§ Designing sensory-substitution devices: Principles, pitfalls and potential1 > Research on similarities and differences between perceptual processing in different modalities
What the sources say
- Supportedcites Deroy, Chen & Spence, 2014 source
The Introduction argues that the paradigmatic case of perceptual awareness is likely to be multisensory and criticizes theories that study awareness one sense at a time. This supports the citing sentence’s broad point that an account of perception needs to consider multisensory interactions; the paper does not assume that those interactions always produce a unified percept. (LLM, conf=0.88)
- Not checked — no full text of the sourcecites Driver & Spence, 2000 source
- Not matchedcites Stein & Meredith, 1993
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
boostedstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: unclear — It is unclear whether it could be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Although this presents many challenges, a feedback system that codes the environment through haptics and audition in conjunction may compensate for the drawbacks of using only each individual sense if strengths are strategically utilized and weaknesses avoided (Shull & Damian, 2015).”
§ Designing sensory-substitution devices: Principles, pitfalls and potential1 > The possibility of combining haptics and audition for sensory substitution
What the sources say
- Partly supportedcites Shull & Damian, 2015 sourceExperimental — accuracy not yet measured
The “Vision aid for the blind” section (page not provided) describes haptic feedback that conveys information about objects and obstacles in the environment. The Conclusions (page not provided) advocate combining multiple haptic modalities, but the copy does not say that jointly coding the environment through haptics and audition compensates for the drawbacks of either sense alone. (LLM, conf=0.83)
hedgedstates a possible cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Neural activity related to multisensory interactions can be seen surprisingly close to cortical areas typically considered unimodal (Driver & Noesselt, 2008).”
§ Designing sensory-substitution devices: Principles, pitfalls and potential1 > Research on similarities and differences between perceptual processing in different modalities
What the sources say
- Supportedcites Driver & Noesselt, 2008 source
The section ‘Different Accounts and Possible Architectures for Multisensory Influences on “Sensory-Specific” Processing’ describes multisensory zones adjacent to sensory-specific cortex and rapid multisensory effects in and around auditory cortex. This supports the claim that such activity occurs close to areas traditionally considered unimodal; no page number is supplied in our copy. (LLM, conf=0.92)
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
86 more claims · 18 with their cited sources checked
Create a free account to see all 144 claims and every source verdict.
- Probably not a substantial claim. 30
- It may state two separate claims as one. 32
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Designing sensory-substitution devices: Principles, pitfalls and potential1 > Neural evidence regarding sensory substitution
“Poirier et al. (2007; see also Renier et al., 2005) show how tactile stimulation can cause activity change in visual areas.”
§ Designing sensory-substitution devices: Principles, pitfalls and potential1 > References
“Poirier C., De Volder A., Tranduy D., & Scheiber C.,. 2007 Pattern recognition using a device substituting audition for vision in blindfolded sighted subjects.”
Citations
Cites
The fusiform face area: a module in human extrastriate cortex specialized for face perception
Kanwisher, N., McDermott, J., & Chun, M. M. (1997) · The Journal of Neuroscience · doi:10.1523/jneurosci.17-11-04302.1997 · Cited by 7,738 · cited 11×
“Furthermore, recruitment of unused visual cortex can help with compensation for the loss of vision (Pasqualotto & Proulx, 2012; Merabet & PascualLeone, 2010).”
BackgroundInfluentialHearing lips and seeing voices
MCGURK, H., & MACDONALD, J. (1976) · Nature · doi:10.1038/264746a0 · Cited by 6,183 · cited 2×
“The McGurk effect is another example of a multimodal illusion where lip movements and heard speech interact (McGurk & MacDonald, 1976).”
BackgroundControlled and automatic human information processing: I. Detection, search, and attention.
Schneider, W., & Shiffrin, R. M. (1977) · Psychological Review · doi:10.1037/0033-295x.84.1.1 · Cited by 5,902 · cited 1×
“Humans can only process a limited amount of the information hitting their senses at a given time, since attentional capacity is severely limited (Broadbent, 1958; Treisman, 1960; Neisser, 1967; Most et al., 2005; Kristjánsson, 2006; Schneider & Shiffrin, 1977).”
BackgroundHumans integrate visual and haptic information in a statistically optimal fashion
Ernst, M. O., & Banks, M. S. (2002) · Nature · doi:10.1038/415429a · Cited by 4,844 · cited 1×
“In another study, Ernst and Banks (2002) observed that while both haptics and vision conveyed accurate information about the height of a bar this was much less reliable for touch than vision.”
Background(2016) · Interpersonal Communication · doi:10.4324/9781315506173-8 · Cited by 4,693
Broadbent, D. E. (1958) · doi:10.1037/10037-000 · Cited by 4,337
PENFIELD, W., & BOLDREY, E. (1937) · Brain · doi:10.1093/brain/60.4.389 · Cited by 4,188 · cited 2×
“They found that cross-modal plasticity developed rapidly and that the occipital cortex was involved in tactile discrimination.”
BackgroundHenry, M. A. (1995) · The Journal of the American Dental Association · doi:10.14219/jada.archive.1995.0108 · Cited by 3,416
The period of susceptibility to the physiological effects of unilateral eye closure in kittens
Hubel, D. H., & Wiesel, T. N. (1970) · The Journal of Physiology · doi:10.1113/jphysiol.1970.sp009022 · Cited by 3,030 · cited 1×
“The brain is capable of considerable reorganization of neural function (Held et al., 2011; Rauschecker, 1995), while younger brains are typically more flexible (Hubel & Wiesel, 1970; Merabet & Pascual-Leone, 2010).”
BackgroundTo See or not to See: The Need for Attention to Perceive Changes in Scenes
Rensink, R. A., O'Regan, J. K., & Clark, J. J. (1997) · Psychological Science · doi:10.1111/j.1467-9280.1997.tb00427.x · Cited by 2,338
Research works printed but not linked: 5
- Grimes, 1996, Vancouver studies in cognitive science: 5. Perception, 89
- Neisser, 1967, Cognitive Psychology.
- Stein, 1993, The Merging of the Senses.
- Weinstein, 1968, The Skin Senses, 195
- Wurtz, 2000, of neural science, 523
Cited by
Human augmentation: Past, present and future
Raisamo, R., Rakkolainen, I., Majaranta, P., Salminen, K., Rantala, J., & Farooq, A. (2019) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2019.05.008 · Cited by 236
“Augmented senses can also enable sensory substitution or sensory prosthesis, in which information from one sense can be mediated through a different sense (see examples of sensory substitution: Kristjánsson et al., 2016; Wright et al., 2015).”
BackgroundEXTENDED REALITY (XR) ENVIRONMENTS
Stanney, K. M., Nye, H., Haddad, S. D., Hale, K. S., Padron, C. K., & Cohn, J. V. (2021) · Wiley eBooks · doi:10.1002/9781119636113.ch30 · Cited by 93
Developments in the human machine interface technologies and their applications: a review
Singh, H. P., & Kumar, P. (2021) · Journal of Medical Engineering & Technology · doi:10.1080/03091902.2021.1936237 · Cited by 73
E-Skins: Biomimetic Sensing and Encoding for Upper Limb Prostheses
Iskarous, M. M., & Thakor, N. V. (2019) · Proceedings of the IEEE · doi:10.1109/jproc.2019.2939369 · Cited by 60
“[17] Á.”
BackgroundInfluentialHoffmann, R., Valgeirsdóttir, V. V., Jóhannesson, Ó. I., Unnþórsson, R., & Kristjansson, A. K. (2018) · Experimental Brain Research · doi:10.1007/s00221-018-5387-z · Cited by 54
Self-citationRelative vibrotactile spatial acuity of the torso
Jóhannesson, Ó. I., Hoffmann, R., Valgeirsdóttir, V. V., Unnþórsson, R., Moldoveanu, A. D. B., & Kristjansson, A. K. (2017) · Experimental Brain Research · doi:10.1007/s00221-017-5073-6 · Cited by 46
“Applications that stimulate passive body parts (torso, arm) may be most practical however, since the tongue and hands should preferably be available for other use (Dakopoulos and Bourbakis 2010; Kristjánsson et al. 2016).”
BackgroundSelf-citationHoffmann, R., Spagnol, S., Kristjánsson, Á., & Unnthorsson, R. (2018) · Optometry and Vision Science · doi:10.1097/opx.0000000000001284 · Cited by 45
“Sensory substitution devices challenge developers and designers because many issues must be considered if the visually impaired are to be expected to use a particular device, for example, usability, portability, comfort, real-time and long-term operation, accessibility of interface, and appearance.(7,8) Sensory substi…”
Self-citationStereo Vision Based Sensory Substitution for the Visually Impaired
Caraiman, S., Zvoristeanu, O., Burlacu, A., & Herghelegiu, P. (2019) · Sensors · doi:10.3390/s19122771 · Cited by 44
“Besides the general limitations of the visual rehabilitation itself, the lack of efficient training programs and tools is considered to be an important factor that hinders the adoption of such SSDs by large communities of visually impaired [42,43].”
BackgroundSupporting Rhythm Activities of Deaf Children using Music-Sensory-Substitution Systems
Petry, B., Illandara, T., Elvitigala, D. S., & Nanayakkara, S. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3174060 · Cited by 41
“[17] stated that sensory-substitution systems always have a learning curve, where steepness depends on the mapping.”
BackgroundNavigation aid for blind persons by visual-to-auditory sensory substitution: A pilot study
Neugebauer, A., Rifai, K., Getzlaff, M., & Wahl, S. (2020) · PLOS ONE · doi:10.1371/journal.pone.0237344 · Cited by 41
“The lack of subtlety and wearing comfort is a problem that many SSDs have not yet overcome and that is seen as a major factor that still prevents blind persons from using SSDs [20, 53, 54].”
BackgroundInfluentialPerception of vibrotactile distance on the back
Plaisier, M. A., Sap, L. I. N., & Kappers, A. M. L. (2020) · Scientific Reports · doi:10.1038/s41598-020-74835-x · Cited by 40
Maćkowski, M., Brzoza, P., Kawulok, M., Meisel, R., & Spińczyk, D. (2023) · ACM Transactions on Multimedia Computing Communications and Applications · doi:10.1145/3586076 · Cited by 45
“[27] points out, it can lead to sensory overload.”
BackgroundThe Role of Visual Factors in Dyslexia
Kristjansson, A. K., & Sigurdardottir, H. M. (2023) · Journal of Cognition · doi:10.5334/joc.287 · Cited by 52
Self-citationIndividual Differences in Sensory Substitution
Arnold, G., Lerousseau, J. P., & Auvray, M. (2017) · Multisensory Research · doi:10.1163/22134808-00002561 · Cited by 31
Current Use and Future Perspectives of Spatial Audio Technologies in Electronic Travel Aids
Spagnol, S., Wersényi, G., Bujacz, M., Balan, O. C., Martínez, M. H., Moldoveanu, A. D. B., & Unnþórsson, R. (2018) · Wireless Communications and Mobile Computing · doi:10.1155/2018/3918284 · Cited by 35
“The auditory outputs of such systems range from simple binary alerts indicating the presence of an obstacle in the range of a sensor, to complex spatial sound patterns aiming at sensory substitution and carrying almost as much information as a graphical image [7, 11].”
BackgroundSelf-citationSoundSight: a mobile sensory substitution device that sonifies colour, distance, and temperature
Hamilton-Fletcher, G., Alvarez, J., Obrist, M., & Ward, J. (2022) · Journal on Multimodal User Interfaces · doi:10.1007/s12193-021-00376-w · Cited by 28
Social Interaction Without Vision: An Assessment of Assistive Technology for The Visually Impaired
Phillips, M., & Proulx, M. J. (2018) · Technology & Innovation · doi:10.21300/20.1-2.2018.85 · Cited by 30
Blind wayfinding with physically-based liquid sounds
Spagnol, S., Hoffmann, R., Martínez, M. H., & Unnþórsson, R. (2018) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2018.02.002 · Cited by 27
Self-citationVibrotactile Threshold Measurements at the Wrist Using Parallel Vibration Actuators
Ævarsson, E. A., Ásgeirsdóttir, T., Pind, F., Kristjansson, A. K., & Unnþórsson, R. (2022) · ACM Transactions on Applied Perception · doi:10.1145/3529259 · Cited by 27
“Sensory substitution builds on the idea that when loss occurs in one sense, it can be augmented through stimulation of another sensory modality [13, 15, 20].”
BackgroundSelf-citationHu, D., Wang, D., Li, X., Nie, F., & Wang, Q. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00816 · Cited by 21
“This probably comes from the participants’ memory of object color [18].”
BackgroundInfluentialTactile spatial discrimination on the torso using vibrotactile and force stimulation
Jouybari, A. F., Franza, M., Kannape, O. A., Hara, M., & Blanke, O. (2021) · Experimental Brain Research · doi:10.1007/s00221-021-06181-x · Cited by 23
“Accordingly, the torso has been considered as one of the most available and practical candidate sites for wearable and mobile tactile communication systems (Cholewiak et al. 2004; Kristjánsson et al. 2016) and may also be particularly suited for applications in cognitive and clinical neurosciences (Rognini and Blanke…”
BackgroundRichardson, M., Thar, J., Alvarez, J., Borchers, J., Ward, J., & Hamilton-Fletcher, G. (2019) · Perception · doi:10.1177/0301006619873194 · Cited by 26
“…as an intervention to assist in daily tasks (Hamilton-Fletcher et al., 2016; Elli, Benetti, & Collignon, 2014; Maidenbaum, Abboud, & Amedi, 2014; Kristjánsson et al., 2016), but it allows users to interpret their remaining senses more accurately (Cappagli et al., 2017), and explore how our…”
BackgroundArtificial referred sensation in upper and lower limb prosthesis users: a systematic review
Gonzalez, M. A., Bismuth, A., Lee, C., Chestek, C. A., & Gates, D. H. (2022) · Journal of Neural Engineering · doi:10.1088/1741-2552/ac8c38 · Cited by 27
“The most common method for providing a prosthesis user with additional sensory feedback is via sensory substitution [2, 8, 9].”
Sensory Substitution for Force Feedback Recovery: A Perception Experimental Study
Avilés-Rivero, A. I., Alsaleh, S. M., Philbeck, J. W., Raventos, S. P., Younes, N., Hahn, J. K., & Casals, A. (2018) · ACM Transactions on Applied Perception · doi:10.1145/3176642 · Cited by 18
“The use of multimodal feedback can be affected in real clinical environments by the attentional capacity of humans, which constrains the amount of information that can be effectively processed [32, 58].”
BackgroundIsaksson, J., Jansson, T. T., & Nilsson, J. (2020) · IEEE Transactions on Neural Systems and Rehabilitation Engineering · doi:10.1109/tnsre.2020.2985616 · Cited by 20
“It is common to, at least partly, attribute the unsuccessful nature of ETAs to inadequate or non-existent training regimes and costly or inaccessible hardware [42], [44], [45].”
BackgroundLupu, R. G., Mitruț, O., Stan, A., Ungureanu, F., Kalimeri, K., & Moldoveanu, A. D. B. (2020) · Sensors · doi:10.3390/s20205821 · Cited by 22
“In the VTE tests, the VIP wore the headset consisting of a structure sensor stereo video camera and an IMU sensor [45,46], but only the IMU signal was used in order to orient in the virtual environment by head movements.”
MethodSelf-citationYeganeh, N., Makarov, I. M., Unnþórsson, R., & Kristjansson, A. K. (2023) · Actuators · doi:10.3390/act12060224 · Cited by 20
Self-citationYoo, M., Odom, W., & Berger, A. (2021) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445212 · Cited by 24
Perception of Tactile Distance on the Back
Nicula, A., & Longo, M. R. (2021) · Perception · doi:10.1177/03010066211025384 · Cited by 20
A Sensory Substitution Device Inspired by the Human Visual System
Osinski, D., & Hjelme, D. R. (2018) · 2018 11th International Conference on Human System Interaction (HSI) · doi:10.1109/hsi.2018.8431078 · Cited by 15
Discrimination Accuracy of Sequential Versus Simultaneous Vibrotactile Stimulation on the Forearm
Yeganeh, N., Makarov, I. M., Kristjansson, A. K., & Unnþórsson, R. (2023) · Applied Sciences · doi:10.3390/app14010043 · Cited by 17
Self-citationHoffmann, R., Brinkhuis, M. A. B., Unnþórsson, R., & Kristjansson, A. K. (2019) · Journal of Neurophysiology · doi:10.1152/jn.00125.2019 · Cited by 18
“Developers of tactile devices may therefore be tempted to mount as many tactors as possible on a tactile device, risking overexertion of the tactile sense (see discussion in Kristjánsson et al. 2016; Dakopoulos and Bourbakis 2010).”
BackgroundMethodSelf-citationA novel, wearable, electronic visual aid to assist those with reduced peripheral vision
Brown, F. E., Sutton, J., Yuen, H. M., Green, D., Dorn, S. V., Braun, T. A., Cree, A. J., Russell, S. R., & Lotery, A. J. (2019) · PLoS ONE · doi:10.1371/journal.pone.0223755 · Cited by 21
Isaksson, J., Jansson, T. T., & Nilsson, J. (2020) · IEEE Transactions on Neural Systems and Rehabilitation Engineering · doi:10.1109/tnsre.2020.2985626 · Cited by 15
“In recent years, with ever-improving technology, the field has been especially active with several reviews outlining proposed electronic travel aids, ETAs, in general [14], [17]–[19]; sensory supplementation/ substitution devices, SSDs, [20], both [21]; and assistive technology for BLVIs in general [22].”
BackgroundSeeing with the Hands: A Sensory Substitution That Supports Manual Interactions
Teng, S.-Y., Kim, G. S.-H., Liu, X., & Lopes, P. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713419 · Cited by 23
“Given the extensive range of this field, readers can refer to the reviews on the subject [5, 15, 36, 40, 62].”
Bordeau, C., Scalvini, F., Migniot, C., Dubois, J., & Ambard, M. (2023) · Frontiers in Psychology · doi:10.3389/fpsyg.2023.1079998 · Cited by 14
“Long trainings are required to master a visual-to-auditory SSD (Kristjánsson et al., 2016) because the used visual-to-auditory encodings are not enough intuitive (Hamilton-Fletcher et al.”
BackgroundDeep Learning-Based Optimization of Visual–Auditory Sensory Substitution
Kim, M., Park, Y., Moon, K., & Jeong, C. Y. (2023) · IEEE Access · doi:10.1109/access.2023.3243641 · Cited by 12
“According to previous studies [6], [7], many factors hinder the widespread use of SSDs: high cost, difficulty of use, difficulty of operation by users with visual impairment and blindness, and the long time required to adapt to these”
BackgroundBerger, C. C., Coppi, S., & Ehrsson, H. H. (2023) · Experimental Brain Research · doi:10.1007/s00221-023-06586-w · Cited by 15
“, the sense of agency) is the result of the agreement between our intentions (Haggard 2017) to act and the sensory consequences of our actions (Engbert et al. 2007; Haggard 2017; Kristjánsson et al. 2016; Lynn et al. 2010; Moore and Obhi 2012) as well as the match between the predicted sensory consequences of movement…”
BackgroundOsiński, D., Łukowska, M., Hjelme, D. R., & Wierzchoń, M. (2021) · Sensors · doi:10.3390/s21217351 · Cited by 11
“…and functionality None-theless, the first prototype had multiple issues regarding its usability and functionality. the previous implementation, we used a standard webcam attached to a appeared to be bulky Firstly, in the previous implementation, we used a standard webcam attached to a head-band,…”
BackgroundMethodInfluentialProulx, M. J., Brown, D. J., Esenkaya, T., Leveson, J. B., Todorov, O. S., Watson, S. H., & de Sousa, A. A. (2020) · Applied Ergonomics · doi:10.1016/j.apergo.2020.103072 · Cited by 12
Karimi, M., Yeganeh, N., Makarov, I. M., Sverrisson, A. Ö., Gunnarsson, K. F., Briem, K., Brynjólfsson, S., Kristjansson, A. K., & Unnþórsson, R. (2025) · Bioengineering · doi:10.3390/bioengineering12090989 · Cited by 18
Self-citationSpagnol, S., Baldan, S., & Unnþórsson, R. (2017) · IEEE International Workshop on Multimedia Signal Processing (MMSP) · doi:10.1109/mmsp.2017.8122220 · Cited by 12
Self-citationMalesevic, J. M., Kostic, M. D., Jure, F. A., Spaich, E. G., Dosen, S. S., Ilić, V., Bijelić, G., & Strbac, M. D. (2022) · Sensors · doi:10.3390/s22197658 · Cited by 11
“In addition, to ensure the hands-free information transfer [47,48], it has been shown that the tactile display placed on the torso can allow high-performance communication when electrode pad size, arrangement and patterns are selected properly [49].”
BackgroundEvaluation of Short-Range Depth Sonifications for Visual-to-Auditory Sensory Substitution
Commère, L., & Rouat, J. (2023) · IEEE Transactions on Human-Machine Systems · doi:10.1109/thms.2023.3265972 · Cited by 10
“Yet, several works underline the importance of choosing effective vision-toaudition mappings for designing such systems [23], [24], [25].”
BackgroundMaćkowski, M., Kawulok, M., Brzoza, P., Janczy, M., & Spińczyk, D. (2023) · Applied Sciences · doi:10.3390/app13179989 · Cited by 13
Maimon, A., Wald, I. Y., Oz, M. B., Codron, S., Netzer, O., Heimler, B., & Amedi, A. (2023) · Frontiers in Human Neuroscience · doi:10.3389/fnhum.2022.1058093 · Cited by 10
“Braille conveys verbal information through haptic or tactile stimulation (Kristjánsson et al., 2016).”
BackgroundInfluentialAnalysis and Validation of Cross-Modal Generative Adversarial Network for Sensory Substitution
Kim, M., Park, Y., Moon, K., & Jeong, C. Y. (2021) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph18126216 · Cited by 8
Yeganeh, N., Makarov, I. M., Thors, S. S. S., Kristjansson, A. K., & Unnþórsson, R. (2022) · Actuators · doi:10.3390/act12010006 · Cited by 9
Self-citationPedzisai, E., & Charamba, S. (2023) · Transportation Research Interdisciplinary Perspectives · doi:10.1016/j.trip.2023.100952 · Cited by 12
Learning and navigating digitally rendered haptic spatial layouts
Tivadar, R. I., Franceschiello, B., Minier, A., & Murray, M. M. (2023) · npj Science of Learning · doi:10.1038/s41539-023-00208-4 · Cited by 14
Łukowska, M., Kałwak, W., Osiński, D., Janik, J., & Wierzchoń, M. (2022) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2022.102925 · Cited by 11
Autoencoding sensory substitution
Tóth, V., & Parkkonen, L. (2019) · Unpublished · doi:10.13140/rg.2.2.10576.87048 · Cited by 7
“cross-modal connections and the receiving modality, we can specifically tailor the soundscapes so they are easily mapped onto the visual cortex, reducing the amount of necessary cross-modal plasticity [27]. The “Holy Grail” [28] of V2A SS is the synaesthetic experience: visual perception consistently and involuntarily…”
BackgroundElectrophysiological Dynamics of Visual-Tactile Temporal Order Perception in Early Deaf Adults
Scurry, A. N., Chifamba, K., & Jiang, F. (2020) · Frontiers in Neuroscience · doi:10.3389/fnins.2020.544472 · Cited by 7
“Not only is visual-tactile temporal acuity important for perceived body representation, improved development of sensory substitution devices relies on understanding the affected person’s perceptual experience, specifically what affects perception of multimodal synchrony (Kristjánsson et al., 2016).”
BackgroundParry, R., Sarlegna, F. R., Jarrassé, N., & Roby-Brami, A. (2021) · Journal of Neurophysiology · doi:10.1152/jn.00517.2020 · Cited by 10
The Haptic Intensity Order Illusion Is Caused by Amplitude Changes
Makarov, I. M., Thors, S. S. S., Ævarsson, E. A., Pind, F., Yeganeh, N., Kristjansson, A. K., & Unnþórsson, R. (2023) · ACM Transactions on Applied Perception · doi:10.1145/3626237 · Cited by 8
Self-citationLocalizing 3D motion through the fingertips: Following in the footsteps of elephants
Snir, A., Cieśla, K., Ozdemir, G., Vekslar, R., & Amedi, A. (2024) · iScience · doi:10.1016/j.isci.2024.109820 · Cited by 6
“33,74,75 Forbothmodalitiestheresultswerealsogenerallylowerwhenthetaskbecamemorecomplexwithverticalmotionaddedtothehorizontal motion (although no actual comparison of performance was possible).”
A Study of Cutaneous Perception Parameters for Designing Haptic Symbols towards Information Transfer
Nyasulu, T. D., Du, S., Steyn, N., & Dong, E. (2021) · Electronics · doi:10.3390/electronics10172147 · Cited by 6
Witter, M., de Rooij, A., van Dartel, M., & Krahmer, E. (2022) · Frontiers in Computer Science · doi:10.3389/fcomp.2022.991180 · Cited by 8
Introducing a New Haptic Illusion to Increase the Perceived Resolution of Tactile Displays
Hoffmann, R., Brinkhuis, M. A. B., Kristjansson, A. K., & Unnþórsson, R. (2018) · International Conference on Computer-Human Interaction Research and Applications (CHIRA) · doi:10.5220/0006899700450053 · Cited by 6
“…high-resolution displays are an especially auspicious choice as hardware components for Sensory Substitution devices (SSDs), non-invasive human-machine interfaces that draw on the central nervous system by bypassing the impaired peripheral components (Kristjánsson et al., 2016; Reich et al., 2012).”
BackgroundSelf-citationInvestigating the Effect of Sensory Concurrency on Learning Haptic Spatiotemporal Signals
Carson, I., Quigley, A. J., Clarke, L., & Hinrichs, U. (2021) · Proceedings of the ACM on Interactive Mobile Wearable and Ubiquitous Technologies · doi:10.1145/3448102 · Cited by 5
“Due to the abstract nature of the information in the signal, there is often a steep learning curve associated with these devices, long-recognized as a barrier to uptake [4, 15, 23, 43].”
BackgroundAngeleska, E., Aleksovska, A., Avramov, N., et al. (2022) · Proceedings of the Design Society · doi:10.1017/pds.2022.206 · Cited by 5
Feeling the music: exploring emotional effects of auditory-tactile musical experiences
Schwartz, N., Snir, A., & Amedi, A. (2025) · Frontiers in Virtual Reality · doi:10.3389/frvir.2025.1592652 · Cited by 4
Shin, S.-Y., Baek, S., Park, S., Yi, H., Shin, H., & Lee, W. (2025) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2025.2452192 · Cited by 5
“In sensory-substitution systems, training is an important consideration when evaluating system improvement (Kristj � ansson et al., 2016).”
Orientation of tactile attention on the surface of the tongue.
Richardson, M., Petrini, K., & Proulx, M. J. (2022) · Journal of Experimental Psychology Human Perception & Performance · doi:10.1037/xhp0001007 · Cited by 6
The role of visual factors in dyslexiaPreprint
Kristjansson, A. K., & Sigurdardottir, H. M. (2022) · doi:10.31234/osf.io/n8xer · Cited by 4
Self-citationIsaksson, J., Jansson, T. T., & Nilsson, J. (2024) · IEEE Access · doi:10.1109/access.2024.3471177 · Cited by 5
Vibrotactile pattern recognition:Influence of interstimulus intervals
Yeganeh, N., Makarov, I. M., Kristjansson, A. K., & Unnþórsson, R. (2025) · Virtual Reality & Intelligent Hardware · doi:10.1016/j.vrih.2025.06.001 · Cited by 5
Self-citationButorovа, A. S., & Sergeev, A. (2025) · The European Physical Journal Special Topics · doi:10.1140/epjs/s11734-025-01596-x · Cited by 3
“…sonification methods in visual-to-auditory SSDs, a fundamental question is how much information from the video camera’s field of view should be converted into an auditory signal, and whether this information should be simply transmitted or significantly preprocessed during the conversion [13, 14].”
Allison, T. S., Moritz, J., Turk, P. J., & Stone-Roy, L. M. (2020) · PLoS ONE · doi:10.1371/journal.pone.0237142 · Cited by 4
“Modern research focuses on the development and testing of specific devices for sensory substitution in an effort to expand available resources for those with sensory impairment [1, 2].”
BackgroundConcerns Regarding Sensory Substitution: Overload and Conflict
Yun, O. T., & Deng, Q. (2022) · Advances in Social Science, Education and Humanities Research/Advances in social science, education and humanities research · doi:10.2991/assehr.k.220401.185 · Cited by 3
Reaching and Grasping With a Sensory Substitution Glove: Variability of Practice Effects
Bilal, S., Bongers, R. M., & Jacobs, D. M. (2025) · Journal of Motor Learning and Development · doi:10.1123/jmld.2024-0060 · Cited by 3
Prototyping and Evaluating Sensory Substitution Devices by Spatial Immersion in Virtual Environments
Guezou‐Philippe, A., Huet, S., Pellerin, D., & Graff, C. (2018) · International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications (VISIGRAPP) · doi:10.5220/0006637705960602 · Cited by 4
“The reader can refer to (Kristjánsson et al., 2016) for a state of the art on SSD.”
BackgroundBizoń-Angov, P., Gwiaździński, P., Hat, K., Osiński, D., Reuter, M., Fauchald, E., Verma, S., Skylstad, T. K., Sorli, S., Thorsen, J. E., et al. (2019) · IEEE International Conference on Development and Learning (ICDL) · doi:10.1109/devlrn.2019.8850696 · Cited by 4
“Colorophone is a novel SS system, which has been developed taking into consideration design principles described in [25].”
MethodMusical expertise, musical style, and visual attention
Jónasson, P., Kristjansson, A. K., & Jóhannesson, Ó. I. (2021) · Psychology of Music · doi:10.1177/0305735620988882 · Cited by 2
Self-citationBilal, S., Rebate, J., Jacobs, D. M., & Sevillano, V. (2025) · Disability and Rehabilitation Assistive Technology · doi:10.1080/17483107.2025.2511982 · Cited by 2
“While previous studies have primarily focused on ssDs as tools for autonomy and mobility [19,37], few have considered their role in facilitating social interactions. issues related to social engagement were repeatedly mentioned in the interviews. examples of these are the appreciation that…”
Exploration strategies influence visual-auditory sensory substitution in a passive guidance setup
Akar, S., Deschamps, L., & Roy, V. (2025) · Acta Psychologica · doi:10.1016/j.actpsy.2025.105619 · Cited by 2
Residual gaze behaviour during navigation in blindness and low vision
Feng, J., Garcia‐Piña, F., Beheshti, M., Hudson, T. E., Seiple, W., & Rizzo, J. (2026) · Disability and Rehabilitation Assistive Technology · doi:10.1080/17483107.2025.2610357 · Cited by 3
“Sensory substitution is inherently slower than the visual channel, even when impaired, because translated signals require additional cognitive processing [16,17].”
Lounis, C., Benejean, M., Hémont, F., Marrast, P., Brock, A., Tiers, H. C. D., Madras, J.-L., & Ceretto, E. (2023) · Springer eBooks · doi:10.1007/978-3-031-28528-8_16 · Cited by 2
Wang, K., Ke, M., Huang, J., & Zou, N. (2026) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3772318.3791768 · Cited by 1
“Multisensory integration is essential for adapting to environmental changes and responding to stimuli efficiently [54].”
Aziz, M. H., & Al-Shamaa, D. (2019) · AIMS Electronics and Electrical Engineering · doi:10.3934/electreng.2019.2.144 · Cited by 2
The Artistic Potential of Tactile Vision Interfaces: A First Look
Ruhl, A., van Dartel, M., & de Rooij, A. (2018) · International Conference on Tangible, Embedded, and Embodied Interaction (TEI) · doi:10.1145/3173225.3173271 · Cited by 2
“Tactile vision interfaces map data from imaging sensors worn on the body to tactile sensory stimulation in realtime [1] and can be used to achieve a wide range of sensory augmentations [6].”
BackgroundRiviere, M.-A., Gay, S. L., & Pissaloux, E. E. (2018) · Lecture notes in computer science · doi:10.1007/978-3-319-94274-2_16 · Cited by 2
Design of a vision substitution vibrotactile vest for the visually impaired
Kalampalikis, D., & Μουστάκας, Κ. (2018) · Hellenic Conference on Artificial Intelligence · doi:10.1145/3200947.3201055 · Cited by 2
Łukowska, M., Osiński, D., Ciupińska, K., Hat, K., Gwiaździński, P., Pin, S. H. D., Kałwak, W., Korczyk, M., Bizoń-Angov, P., Reuter, M., et al. (2022) · SSRN Electronic Journal · doi:10.2139/ssrn.4113324 · Cited by 3
HapticVLM: VLM-Driven Texture Recognition Aimed at Intelligent Haptic Interaction
Khan, M. H., Cabrera, M. A., Iarchuk, D., Mahmoud, Y., Trinitatova, D., Tokmurziyev, I., & Tsetserukou, D. (2025) · Conference proceedings/Conference proceedings - IEEE International Conference on Systems, Man, and Cybernetics · doi:10.1109/smc58881.2025.11343053 · Cited by 3
“The system establishes a foundation for next-generation haptic technologies, enabling more immersive and perceptually coherent experiences beyond traditional tactile feedback.”
Guo, X.-F., & Wang, S. (2020) · European Journal of Horticultural Science · doi:10.17660/ejhs.2020/85.6.2 · Cited by 1
Valgeirsdóttir, V. V., Sigurðardóttir, J. S., Jóhannesson, Ó. I., Alexandersson, Á., & Kristjansson, A. K. (2021) · JPO Journal of Prosthetics and Orthotics · doi:10.1097/jpo.0000000000000354 · Cited by 1
“Several criteria must be met for beneftits of neuroprosthetic devices to reach users, which resemble requirements reported for solutions intended to increase mobility in other populations.(39) First, the system must be consistent to ensure safety and promote the user's trust.”
BackgroundSelf-citationCommère, L., & Rouat, J. (2022) · IEEE Transactions on Human-Machine Systems · doi:10.1109/thms.2022.3194715 · Cited by 3
Mono vs Stereo Modes of Visual-Auditory Sensory Substitution in Spatial Localization Task
Butorovа, A. S., Koryukin, E. A., Nurullina, N. I., Bobakov, V. S., & Sergeev, A. (2023) · International Conference Neurotechnologies and Neurointerfaces (CNN) · doi:10.1109/cnn59923.2023.10275142 · Cited by 1
Sensory Substitution for Visual Impairments: A Technological Review
Billah, M. M., Yusof, Z. M., Kadir, K., & Ali, A. M. M. (2019) · IntechOpen eBooks · doi:10.5772/intechopen.89147 · Cited by 1
Colorophone 2.2 – A Spatial, Real-Time Color-to-Sound Sensory Substitution DevicePreprint
Osiński, D., Łukowska, M., Kałwak, W., Wierzchoń, M., & Hjelme, D. R. (2022) · doi:10.36227/techrxiv.21385140 · Cited by 1
Improving Healthcare Practice by Using HMI Interface
Verma, V., Dave, V., & Wal, P. (2023) · Wiley eBooks · doi:10.1002/9781394200344.ch2 · Cited by 1
Colorophone 2.2 – A Spatial, Real-Time Color-to-Sound Sensory Substitution DevicePreprint
Osiński, D., Łukowska, M., Kałwak, W., Wierzchoń, M., & Hjelme, D. R. (2024) · doi:10.36227/techrxiv.21385140.v2 · Cited by 2
Human Factors in Virtual Reality
Burdea, G. C., & Coiffet, P. (2024) · Wiley eBooks · doi:10.1002/9781119512608.ch8
Estimating and understanding motion : from diagnostic to robotic surgery
Avilés-Rivero, A. I. (2017) · UPCommons institutional repository (Universitat Politècnica de Catalunya) · doi:10.5821/dissertation-2117-116820
A Time-Discrete Haptic Feedback System for Use by Persons with Lower-Limb Prostheses During Gait
Kaplan, G., McDaniel, T. L., Abbas, J. J., Tadayon, R., & Panchanathan, S. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-23563-5_21 · Cited by 1
Learning and Navigating Digitally-Rendered Haptic Spatial LayoutsPreprint
Tivadar, R. I., Franceschiello, B., Minier, A., & Murray, M. M. (2022) · Research Square · doi:10.21203/rs.3.rs-1874489/v1
Colorophone 2Preprint
Osiński, D., Łukowska, M., Kałwak, W., Wierzchoń, M., & Hjelme, D. R. (2022) · doi:10.36227/techrxiv.21385140.v1
Guiding the Hand to an Invisible Target
Furtak, M., & Brenner, E. (2025) · Journal of Motor Behavior · doi:10.1080/00222895.2025.2455179
“Providing too little information might prevent people from achieving the full potential benefit of such a device (Kristj (cid:2) ansson et al., 2016).”
Sensory Cues in Assistive Technologies: a Comparative Study of the BuzzClip and NOAPreprint
Pittet, C. E., & Fabien, M. (2025) · medRxiv · doi:10.1101/2025.02.20.25322389
Metadata
- Authors
- Kristjánsson, Á., Moldoveanu, A., Jóhannesson, Ó. I., Balan, O., Spagnol, S., Valgeirsdóttir, V. V., & Unnthorsson, R.
- Affiliations
- Laboratory of Visual Perception and Visuomotor control, University of Iceland, Faculty of Psychology, School of Health Sciences, Reykjavik, Iceland
- Universitatea Națională de Știință și Tehnologie Politehnica București (RO)
- University of Iceland (IS)
- Faculty of Industrial Engineering, Mechanical Engineering and Computer Science, School of Engineering and Natural Sciences, Reykjavik, Iceland
- Venue
- Restorative Neurology and Neuroscience
- Publisher
- SAGE Publications
- Published
- 2016-09-21
- Volume
- 34
- Issue
- 5
- Pages
- 769-787
- ISSN
- 0922-6028 (ISSN-L)
- 1878-3627
- PMC
- PMC5044782
- OpenAlex
- W2510919644
- Cited by
- 116
- References
- 229
- Citations by year
- 2026: 13 · 2025: 14 · 2024: 6 · 2023: 16 · 2022: 14 · 2021: 14 · 2020: 9 · 2019: 12 · 2018: 14 · 2017: 4
- FWCI
- 1.07
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Restorative Neurology and Neuroscience (publishedVersion, cc-by-nc)
- PubMed Central (publishedVersion, cc-by-nc)
- PubMed (publishedVersion)
- AirIuav (Università Iuav di Venezia) (submittedVersion, other-oa)
- Opin vísindi (Opin vísindi) (submittedVersion, cc-by-nc)
- Funders
- European Commission (HORIZON2020, 643636)
- Topics
- Life Sciences › Neuroscience › Cognitive Neuroscience › Tactile and Sensory Interactions
- Health Sciences › Health Professions › Occupational Therapy › Assistive Technology in Communication and Mobility
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Multisensory perception and integration
- Keywords
- sensory substitution, blind people, sensory deprivation, sensory overload, attentional capacity, spatiotemporal continuity, multisensory integration, device training, externalization, usability
- MeSH terms
- Biofeedback, Psychology, Humans, Perception, Psychophysics, Sensation Disorders, Sensory Aids, Neuroimaging
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.