Visual-vestibular cue integration for heading perception: applications of optimal cue integration theory

“However, in many cases the prior is assumed to be approximately uniform (e.g., Ernst & Banks, 2002; Alais & Burr, 2004; Hillis et al., 2004).”

“…estimate is simply the sum of the individual cue reliabilities: 1 r2AB ¼ 1 r2A þ 1 r2B ; or r2AB ¼ r2Ar 2 B r2A þ r2B ð4Þ A distinct but related approach uses Bayes’ rule to define optimal cue integration (for reviews see Yuille & Bülthoff, 1996; Mamassian et al., 2002; Knill & Pouget, 2004).”

“…for behavioral performance by incorporating top-down expectations or biases about the stimuli, whether they are learned during the course of a task (Körding & Wolpert, 2004) or otherwise established during development or through life experience (e.g., Weiss et al., 2002; Stocker & Simoncelli,…”

“To quantify neuronal sensitivity in a manner similar to behavior, we used signal detection theory (receiver operating characteristic (ROC) analysis; Green & Swets, 1966; Bradley et al., 1987; Britten et al., 1992) to generate neurometric functions for each cell in the three stimulus conditions.”

“However, in many cases the prior is assumed to be approximately uniform (e.g., Ernst & Banks, 2002; Alais & Burr, 2004; Hillis et al., 2004).”

“Vection refers to an illusory percept of self-motion induced by visual motion (Mach, 1875; Brandt et al., 1972; Berthoz et al., 1975; Dichgans & Brandt, 1978; Howard, 1982).”

“The behavioral and perceptual benefits of multisensory integration have been shown to be quite impressive and to confer a highly degree of adaptive benefit (Calvert et al., 2004; Fetsch et al., 2010; Murray and Wallace, 2012; Stein and Meredith, 1993; Stevenson et al., 2014a).”

“, vision (Fetsch et al., 2010), audition, somatosensation (Lopez et al.”

“, the visual-inertial SFM problem (from now on the Vi-SfM problem), has particular interest and has been investigated by many disciplines, both in the framework of computer science Bryson and Sukkarieh (2008), Jones and Soatto (2011), Kelly and Sukhatme (2011), Martinelli (2012), Strelow and Singh (2004) and in the fr…”

“Several results suggest that a process based on dynamic sensory reweighting may be important to explain cybersickness and adaptation to virtual environments (Butler et al. 2010; de Winkel et al. 2010; Fetsch et al. 2010; Angelaki et al. 2011).”

“Previous studies have suggested a role for MSTd in multisensory heading perception (for reviews, see Angelaki et al., 2009; 2011; Fetsch et al., 2010).”

“However, the most thoroughly studied area is the dorsal medial superior temporal area (MSTd) (Britten, 2008; Angelaki et al., 2009; Fetsch et al., 2010).”

“Near-optimalBayesian integration of visual and vestibular information has been measured in several studies (Gu et al. 2008; Butler et al. 2010; Fetsch et al. 2009; Prsa et al. 2012; Drugowitsch et al. 2014; Fetsch et al. 2010).”

“Neural selectivity for heading has been described extensively in the dorsal medial superior temporal (MSTd) area, where many neurons show heading tuning in response to both inertial motion and optic flow (Duffy, 1998; Bremmer et al., 1999; Gu et al., 2006; Britten, 2008; Fetsch et al., 2010; Angelaki et al., 2011).”

“…neurons in MSTd respond during vestibular stimulation in the dark and can display the same or opposite tuning for direction of motion in both visual and vestibular modalities, suggesting multimodal interactions in encoding heading (Bremmer et al., 1999; Gu et al., 2006; Fetsch et al., 2007, 2010).”

“The Bayesian framework has proven extremely successful in accounting for the behavioral (Fetsch et al., 2010) and single-cell recording (Gu et al.”

“, the weight of each cue is proportional to its reliability (1/variance) (171).”

“…some earlier threshold studies have investigated optimal integration by degrading the precision of one or more of the cues (Ernst and Banks 2002; Fetsch et al. 2010), we chose to vary frequency as for our earlier threshold study that investigated optimal integration of visual and vestibular…”

“Therefore, vestibular influences on somatosensation do not seem to act as an additional informative input contributing to multisensory integration (Fetsch et al., 2010).”

“…2010; Beierholm et al., 2009; Besson et al., 2010; Charbonneau et al., 2013; Clemens et al., 2011; Deneve and Pouget, 2004; Ernst and Banks, 2002; Fetsch et al., 2010, 2012, 2009; Helbig et al., 2012; Ma and Pouget, 2008; Nath and Beauchamp, 2011; Noppeney et al., 2010; Rohe and Noppeney, 2015).”

“…numerosity judgment (Shams et al., 2005; Bresciani et al., 2006; Wozny et al., 2008), rate perception (Roach et al., 2006), oddity perception (Fetsch et al., 2009, 2010; Butler et al., 2010), angular displacement (Jürgens and Becker, 2006), gravitoinertial force discrimination (Macneilage et…”

“…by simultaneous sounds (Alais and Burr, 2004; Beer and Roder, 2004; Lewis and Noppeney, 2010; Schmiedchen et al., 2012) or vestibular information (Fetsch et al., 2010; Gu et al., 2008), even so when the multisensory stimulus is not directly task relevant (Gleiss and Kayser, 2014b; Kim et al.,…”

“Visuo-vestibular cue integration is essential for self-motion, motor control and spatial orientation and hence for the interaction with our environment (for a review see Fetsch et al., 2010) and the demarcation of self from non-self (Lopez, 2013).”

“(Duffy 1998; Gu et al. 2006; Fetsch et al. 2010), a recent inactivation study in macaques provided causal evidence that MSTd is dominant for visual heading direction thresholds; VPS and parieto-insular vestibular cortex (PIVC) areas are dominant for vestibular signals, but inactivation of VPS had no functional consequ…”

“…flow and vestibular signals related to self-translation occurs in both the dorsal medial superior temporal area (MSTd; Gu et al., 2006, 2008, 2012; Fetsch et al., 2007, 2009, 2010, 2012; Chen et al., 2008; Yang et al., 2011) and the ventral intraparietal area (VIP; Bremmer et al., 2002a,b, 2005,…”

“The cortical area currently considered the most likely site of convergence of visual-vestibular heading information is the dorsal medial superior temporal area (MSTd) [6, 7], as neurons at this site have been shown to respond to optic flow patterns suggesting self-motion [8–14] as well as to translation in absence of…”

“…i.e., the visual-inertial structure from motion problem (from now on the Vi-SfM problem), has particular interest and has been investigated by many disciplines, both in the framework of computer science [2], [12], [13], [20], [23] and in the framework of neuroscience (e.g., [1], [5], [8]).”

“Most of the studies that have addressed walking in blindfolded subjects, however, have been focused on pathintegration or orientation towards a visual target presented prior to initiating movement, or reproduction of a recently performed route [33,34,35,36].”

“…integration has been studied by providing motion and other sensory cues that are either congruent or conflicting and determining how much weight the brain places on each cue to form an overall estimate of head orientation or movement in the environment (Fetsch et al. 2010; McCollum et al. 1996).”

“…levels (e.g., Alais & Burr, 2004; Battaglia et al., 2003; Butler et al., 2010; Butler et al., 2011; Camors et al., 2015; Ernst & Banks, 2002; Girshick & Banks, 2009; Hillis et al., 2004; Jacobs, 1999; Koppen et al., 2019) and neural levels (e.g., Fetsch et al., 2010, 2013) during perceptual tasks.”

“This involves monitoring sensory flow (mostly from proprioception, vision, and audition) to update rotation and translation of the body relative to the starting point [17].”

“These novel findings extend previous demonstrations that vestibular cues pertaining to self-motion perception are processed according to statistical optimality (Butler et al. 2010; Fetsch et al. 2010; Laurens and Droulez 2007; MacNeilage et al. 2008; Prsa et al. 2012; Vingerhoets et al. 2009).”

“…a number of studies suggest that the brain optimally processes this information (Borah et al. 1988; Butler et al. 2010; de Winkel et al. 2010; Fetsch et al. 2009, 2010, 2011; Gu et al. 2008; Jurgens and Becker 2006; Karmali et al. 2014; Karmali and Merfeld 2012; Laurens and Droulez 2007;…”

“…involves a complex process that relies on the optimal integration of information from the visual, vestibular, proprioceptive, and auditory modalities (Angelaki and Cullen 2008; Chang, Uchanski, and Hullar 2012; Fetsch et al. 2009; Fetsch, DeAngelis, and Angelaki 2010; Lord and Sturnieks 2005).”

“Much of the research that is currently available which addresses how sensory cues are integrated to inform heading perception has been conducted in primates (Gu et al., 2007; Morgan et al., 2008; Angelaki et al., 2009; Fetsch et al., 2010; Chen et al., 2011; Cullen, 2012; Fetsch et al., 2013; Smith et al., 2017).”

“In order to optimally combine different information from across these sensory systems, the reliability of the inputs must be ascertained (Cheng et al., 2007; Deneve and Pouget, 2004; Fetsch et al., 2010).”

“…et al., 2014;Wystrach et al., 2015; Fleischmann et al., 2016) and evidence from landmark navigation studies in pigeons (Sutton, 2002; Blaisdell and Cook, 2005; Cheng et al., 2006) or cue integration inmonkeys (Fetsch et al., 2010; Dokka et al., 2015) also indicates close to optimal cue integration.”

“This problem has been investigated by many disciplines, both in the framework of computer science [7], [22], [23], [29], [34] and in the framework of neuroscience (e.g., [5], [10], [11]).”

“These heading thresholds are comparable to those found for both macaques and humans in previous studies (Gu et al., 2007, 2008; Fetsch et al., 2010; Chen et al., 2013).”

“[51] Fetsch CR, DeAngelis GC, Angelaki DE.”

“For vertebrates, several senses are available to detect self-motion, notably vestibular and visual systems (1).”

“…this correspondence, which is in line with previously reported feature fusion mechanisms observed within and between visual, somato-sensory, vestibular, and auditory sensory modalities (Treisman and Gelade, 1980; Angelaki et al., 2009; Fetsch et al., 2010; Parise et al., 2012; Ernst et al., 2013).”

“This also stands in a visuo-vestibular heading discrimination task [20,64], which is embodied by steeper psychometric curve in the combined condition compared with that in either single-cue condition, as predicted by equation (3.4).”

“When cues are available, combining them facilitates the detection of salient events and reduces perceptual uncertainty, in order to improve reactions to immediate dangers and to rapidly varying events [2].”

“In this issue Fetsch, DeAngelis and Angelaki describe how Bayesian decision theory accounts for the contributions of visual and vestibular information to estimations of one’s translation in space (Fetsch et al., 2010).”

“When visual and vestibular cues are congruent and redundant (as is typically the case during most everyday experiences), integrating these inputs improves the precision of perceptual estimates (Ernst and Bülthoff, 2004; Fetsch et al., 2010; Butler et al., 2015).”

“Particularly, it has been largely explained by the theory of optimal integration (Fetsch et al, 2010, 2013; ter Horst et al., 2015).”

“These multiple sensory cues are integrated to reduce HD errors (Telford et al., 1995; Becker et al., 2002; Fetsch et al., 2010; Frissen et al., 2011).”

“The visual-inertial structure from motion problem (from now on the Vi-SfM problem), has particular interest and has been investigated by many disciplines, both in the framework of computer science [3], [10], [11], [15], [17], [21] and in the framework of neuroscience (e.g., [2], [4], [5]).”

“In such situations, the visual and inertial stimuli are integrated near optimally based on their relative reliabilities or Bayesian ideal (Knill and Pouget 2004; Fetsch et al. 2010; Crane 2017).”

“This filtering-out of a degraded signal has been shown during visuo-vestibular integration for monkey heading perception ( Fetsch et al., 2010 ), or for human object discrimination using visual and haptic senses ( Ernst and Banks, 2002 ).”

“Verticality perception is crucial for postural stability, spatial awareness, and precise motor coordination (Lacquaniti et al. 2014; White et al. 2020; Glasauer and Mittelstaedt 1998; Fetsch et al. 2010; Kheradmand and Winnick 2017).”

“We found that intra-auditory integration enhanced both online and offline motor synergy.”

“Since IN involves a physical motion and CE does not, one could raise the point that PSE measurements are actually measuring visual-vestibular integration, and not vection which is purely visual [42,43].”

“The third hypothesis states that perceived motion is based on an optimal weighted average of visual and vestibular sensory inputs (Fetsch et al., 2010).”

“According to sensory reweighting (Asslander and Peterka 2014 ; Peterka 2002 ) theory, and an optimal cue integration framework, the nervous system down-weights unreliable sensory channels and up-weights the most pre - cise channels (Ernst and Banks 2002 ; Fetsch et al. 2010 ).”