Interactive Multimodal Biofeedback for Task-Oriented Neural Rehabilitation

In rehabilitation, patients performed smoother hand movements when given augmented visual feedback (a semitransparent cone and line showing arm deviation from target) combined with auditory feedback.

“The patients performed smoother hand movements while being provided with the augmented visual (in combination with auditory) feedback (Huang et al., 2005).”

A complex real-time sonification for arm movement mapped arm position to harmonics, smoothness to rhythm, and successful and compensatory movements to auditory alarms, with compensation severity mapped to loudness.

“Harmonics were used to represent arm position, rhythm represented the smoothness of the movement, and auditory alarms were used to indicate a successful movement, as well as a compensatory movement (the severity of the compensation was mapped to loudness) (Chen et al., 2006;Huang et al., 2005).”

Augmented information should be displayed in the appropriate modality or multimodally, because imperfect estimates from one modality can be improved by more precise information in another modality.

“Therefore, it is suggested that augmented information should be displayed in the appropriate modality (Huang et al., 2005), according to the modality appropriateness hypothesis (Welch & Warren, 1980), or in a multimodal way, since imperfect estimations gained by one modality can be improved by more precise information in another modality (Hecht & Reiner, 2009;van Beers, Sittig, & Gon, 1999).”

In a reaching task, representing kinematic variables each in a unimodal way, either visually or by sonification, produced an engaging multimodal feedback design that enhanced performance.

“Therewith, an engaging, multimodal feedback design was present that enhanced performance during the reaching task (Chen et al., 2006;Huang et al., 2005;Wallis et al., 2007).”