Visual-vestibular cue integration for heading perception: applications of optimal cue integration theory

“However, in many cases the prior is assumed to be approximately uniform (e.g., Ernst & Banks, 2002; Alais & Burr, 2004; Hillis et al., 2004).”

“…estimate is simply the sum of the individual cue reliabilities: 1 r2AB ¼ 1 r2A þ 1 r2B ; or r2AB ¼ r2Ar 2 B r2A þ r2B ð4Þ A distinct but related approach uses Bayes’ rule to define optimal cue integration (for reviews see Yuille & Bülthoff, 1996; Mamassian et al., 2002; Knill & Pouget, 2004).”

“…for behavioral performance by incorporating top-down expectations or biases about the stimuli, whether they are learned during the course of a task (Körding & Wolpert, 2004) or otherwise established during development or through life experience (e.g., Weiss et al., 2002; Stocker & Simoncelli,…”

“To quantify neuronal sensitivity in a manner similar to behavior, we used signal detection theory (receiver operating characteristic (ROC) analysis; Green & Swets, 1966; Bradley et al., 1987; Britten et al., 1992) to generate neurometric functions for each cell in the three stimulus conditions.”

“However, in many cases the prior is assumed to be approximately uniform (e.g., Ernst & Banks, 2002; Alais & Burr, 2004; Hillis et al., 2004).”

“Vection refers to an illusory percept of self-motion induced by visual motion (Mach, 1875; Brandt et al., 1972; Berthoz et al., 1975; Dichgans & Brandt, 1978; Howard, 1982).”

“…behavioral performance by incorporating top-down expectations or biases about the stimuli, whether they are learned during the course of a task (Körding & Wolpert, 2004) or otherwise established during development or through life experience (e.g., Weiss et al., 2002; Stocker & Simoncelli, 2006).”

“One of the simplest of such models (Cochran, 1937; Clark & Yuille, 1990; Landy et al., 1995) posits that the lowest-variance unbiased estimate (denoted S), given a set of n sensory cues, is one that takes a weighted linear sum of each single-cue estimate (Si): Ŝ ¼ Xn i¼1 wiŜi ð1Þ The weights wi…”

“Somatosensory and proprioceptive cues, in addition to efference copy signals, are especially critical for the guidance of active movements (Mittelstaedt & Mittelstaedt, 2001; Sun et al., 2004; Angelaki & Cullen, 2008) which comprise most natural self-motion.”

“MSTd and VIP neurons have large receptive fields and directional selectivity for optic flow patterns that simulate self-motion (Tanaka et al., 1986; Tanaka & Saito, 1989; Duffy & Wurtz, 1991, 1995; Schaafsma & Duysens, 1996; Bremmer et al., 2002).”

“Second, the task is well suited for using tools developed in classic studies of visual motion discrimination (Britten et al., 1992; Shadlen et al., 1996), including a principled method for manipulating cue reliability (motion coherence), and the use of signal-detection theory to compare neuronal…”

“Vestibular research has lagged behind other sensory systems, particularly vision and somatosensation (Parker & Newsome, 1998; Luna et al., 2005), in terms of understanding the information contained in neuronal responses, the uncertainty associated with that information, and how signals might be…”

“Since the pioneering work of James Gibson (1950, 1954, 1958) it has been widely recognized that visual cues provide a rich source of information about our movements through the environment.”

“…behavioral performance by incorporating top-down expectations or biases about the stimuli, whether they are learned during the course of a task (Körding & Wolpert, 2004) or otherwise established during development or through life experience (e.g., Weiss et al., 2002; Stocker & Simoncelli, 2006).”

“…has been successfully applied in a number of theoretical and psychophysical studies of cue integration (e.g., Landy et al., 1995; Jacobs, 1999; van Beers et al., 1999; Landy & Kojima, 2001; Ernst & Banks, 2002; van Beers et al., 2002; Knill & Saunders, 2003; Oruc et al., 2003; Alais & Burr, 2004;…”

“…Ltd European Journal of Neuroscience, 31, 1721–1729 MSTd and VIP neurons have large receptive fields and directional selectivity for optic flow patterns that simulate self-motion (Tanaka et al., 1986; Tanaka & Saito, 1989; Duffy & Wurtz, 1991, 1995; Schaafsma & Duysens, 1996; Bremmer et al., 2002).”

“…proprioceptive and motor-related signals as early as one or two synapses from the periphery, in the brainstem vestibular nuclei (Henn et al., 1974; Robinson, 1977; Waespe & Henn, 1977; Daunton & Thomsen, 1979) and vestibulocerebellum (Waespe et al., 1981; Waespe & Henn, 1981; Markert et al., 1988).”

“…studies of cue integration (e.g., Landy et al., 1995; Jacobs, 1999; van Beers et al., 1999; Landy & Kojima, 2001; Ernst & Banks, 2002; van Beers et al., 2002; Knill & Saunders, 2003; Oruc et al., 2003; Alais & Burr, 2004; Hillis et al., 2004; Jürgens & Becker, 2006; MacNeilage et al.,…”

“Heading sensitivity of MSTd neurons (Gu et al., 2008).”

“…of cue reliability (uncertainty) exists in the responses of sensory neurons, an important follow-up question is whether and how this representation is decoded into a more explicit form (e.g., Jazayeri & Movshon, 2006; Jazayeri, 2008) that can be used to guide decisions and predict their outcomes.”

“One of the simplest of such models (Cochran, 1937; Clark & Yuille, 1990; Landy et al., 1995) posits that the lowest-variance unbiased estimate (denoted S), given a set of n sensory cues, is one that takes a weighted linear sum of each single-cue estimate (Si): Ŝ ¼ Xn i¼1 wiŜi ð1Þ The weights wi…”

“…inability to distinguish translational motion from changes in orientation relative to gravity (the tilt–translation ambiguity; Angelaki et al., 1999; Merfeld et al., 1999; Angelaki et al., 2004), but in fact this ambiguity is resolved in the brain by incorporating angular velocity signals from…”

“One of the simplest of such models (Cochran, 1937; Clark & Yuille, 1990; Landy et al., 1995) posits that the lowest-variance unbiased estimate (denoted S), given a set of n sensory cues, is one that takes a weighted linear sum of each single-cue estimate (Si): Ŝ ¼ Xn i¼1 wiŜi ð1Þ The weights wi…”

“Vection refers to an illusory percept of self-motion induced by visual motion (Mach, 1875; Brandt et al., 1972; Berthoz et al., 1975; Dichgans & Brandt, 1978; Howard, 1982).”

“information with inertial signals that specify the motion of the head in space, such as from the vestibular otolith organs (Guedry, 1974; Fernandez & Goldberg, 1976a, 1976b; Benson et al., 1986).”

“A large body of work has been devoted to understanding the mechanisms by which the brain might compensate for these disruptions of the optic flow field and recover the true heading (for reviews see Lappe et al., 1999; Warren, 2003; Britten, 2008).”

“However, in many cases the prior is assumed to be approximately uniform (e.g., Ernst & Banks, 2002; Alais & Burr, 2004; Hillis et al., 2004).”

“Nevertheless, the most critical observation is that all subjects show clear changes in cue weights as a function of cue reliability, indicating that both monkeys and humans dynamically re-weight visual and vestibular cues on a trial-by-trial basis during self-motion perception (Fetsch et al., 2009).”

“…cue integration (e.g., Landy et al., 1995; Jacobs, 1999; van Beers et al., 1999; Landy & Kojima, 2001; Ernst & Banks, 2002; van Beers et al., 2002; Knill & Saunders, 2003; Oruc et al., 2003; Alais & Burr, 2004; Hillis et al., 2004; Jürgens & Becker, 2006; MacNeilage et al., 2007; Burr et al.,…”

“Similarly, areas traditionally recognized as ‘vestibular cortex’ (e.g., area 2v and parieto-insular vestibular cortex; Schwarz & Fredrickson, 1971; Grüsser et al., 1990; Fukushima, 1997) are largely unresponsive to the types of optic flow stimuli used in the behavioral experiments described above…”

“…translating observers regularly undergo movements of their eyes and head (Royden et al., 1992, 1994; Banks et al., 1996; Crowell et al., 1998), and may also experience motion of other objects in the scene (Gibson, 1954; Warren & Saunders, 1995; Royden & Hildreth, 1996; Warren & Rushton, 2009).”

“…with visual, proprioceptive and motor-related signals as early as one or two synapses from the periphery, in the brainstem vestibular nuclei (Henn et al., 1974; Robinson, 1977; Waespe & Henn, 1977; Daunton & Thomsen, 1979) and vestibulocerebellum (Waespe et al., 1981; Waespe & Henn, 1981;…”

“…Ltd European Journal of Neuroscience, 31, 1721–1729 MSTd and VIP neurons have large receptive fields and directional selectivity for optic flow patterns that simulate self-motion (Tanaka et al., 1986; Tanaka & Saito, 1989; Duffy & Wurtz, 1991, 1995; Schaafsma & Duysens, 1996; Bremmer et al., 2002).”

“…by the work of Ma et al. (2006, 2008) is that cue reliability, perhaps in the form of a probability distribution over stimuli, is encoded implicitly by the activity of sensory neuronal populations (Foldiak, 1993; Sanger, 1996; Deneve et al., 2001; Knill & Pouget, 2004; Ma et al., 2006, 2008).”

“…Ltd European Journal of Neuroscience, 31, 1721–1729 MSTd and VIP neurons have large receptive fields and directional selectivity for optic flow patterns that simulate self-motion (Tanaka et al., 1986; Tanaka & Saito, 1989; Duffy & Wurtz, 1991, 1995; Schaafsma & Duysens, 1996; Bremmer et al., 2002).”

“However, the real world is not so simple: translating observers regularly undergo movements of their eyes and head (Royden et al., 1992, 1994; Banks et al., 1996; Crowell et al., 1998), and may also experience motion of other objects in the scene (Gibson, 1954; Warren & Saunders, 1995; Royden &…”

“studies of cue integration (e.g., Landy et al., 1995; Jacobs, 1999; van Beers et al., 1999; Landy & Kojima, 2001; Ernst & Banks, 2002; van Beers et al., 2002; Knill & Saunders, 2003; Oruc et al., 2003; Alais & Burr, 2004; Hillis et al., 2004; Jürgens & Becker, 2006; MacNeilage et al., 2007; Burr et al., 2009; Ma et al…”

“heading perception might seem to be limited by their inability to distinguish translational motion from changes in orientation relative to gravity (the tilt–translation ambiguity; Angelaki et al., 1999; Merfeld et al., 1999; Angelaki et al., 2004), but in fact this ambiguity is resolved in the brain by incorporating a…”

“Furthermore, the temporal dynamics of vestibular responses in MSTd are generally consistent with the encoding of stimulus velocity (Gu et al., 2006; Liu & Angelaki, 2009), similar to the dynamics of visual motion responses.”

“Note that previous labyrinthectomy experiments (Gu et al., 2007) demonstrated severe and long-lasting performance deficits in the ‘vestibular’ condition, suggesting that the cues used by the animals in this condition were primarily of vestibular origin (rather than somatosensory or proprioceptive).”

“…to be limited by their inability to distinguish translational motion from changes in orientation relative to gravity (the tilt–translation ambiguity; Angelaki et al., 1999; Merfeld et al., 1999; Angelaki et al., 2004), but in fact this ambiguity is resolved in the brain by incorporating angular…”

“Of particular relevance here, at least 50-60% of MSTd neurons are also selective for physical translation in darkness (Duffy, 1998; Bremmer et al., 1999; Gu et al., 2006).”

“However, other limitations of the vestibular system, such as its failure to encode sustained constantvelocity motion (Fernandez & Goldberg, 1976a; Büttner & Waespe, 1981; Si et al., 1997), suggest that the nervous system cannot rely solely on vestibular signals for heading perception, just as it…”

“tions of the optic flow field and recover the true heading (for reviews see Lappe et al., 1999; Warren, 2003; Britten, 2008).”

“…et al., 2002; Knill & Saunders, 2003; Oruc et al., 2003; Alais & Burr, 2004; Hillis et al., 2004; Jürgens & Becker, 2006; MacNeilage et al., 2007; Burr et al., 2009; Ma et al., 2009), each grounded in mathematical principles borrowed from ideal observer analysis, signal detection theory and ⁄…”

“Vestibular research has lagged behind other sensory systems, particularly vision and somatosensation (Parker & Newsome, 1998; Luna et al., 2005), in terms of understanding the information contained in neuronal responses, the uncertainty associated with that information, and how signals might be…”

“This type of experimental design, sometimes known as a perturbation analysis, has been used previously in several human psychophysical studies of cue integration in other sensory modalities (e.g., Young et al., 1993; Landy et al., 1995; Ernst & Banks, 2002; Alais & Burr, 2004).”

“…Ltd European Journal of Neuroscience, 31, 1721–1729 MSTd and VIP neurons have large receptive fields and directional selectivity for optic flow patterns that simulate self-motion (Tanaka et al., 1986; Tanaka & Saito, 1989; Duffy & Wurtz, 1991, 1995; Schaafsma & Duysens, 1996; Bremmer et al., 2002).”

“In the model of Zacharias & Young (1981), for example, it was proposed that visual and vestibular cues combine in a weighted fashion to estimate self-rotation velocity.”

“studies of cue integration (e.g., Landy et al., 1995; Jacobs, 1999; van Beers et al., 1999; Landy & Kojima, 2001; Ernst & Banks, 2002; van Beers et al., 2002; Knill & Saunders, 2003; Oruc et al., 2003; Alais & Burr, 2004; Hillis et al., 2004; Jürgens & Becker, 2006; MacNeilage et al., 2007; Burr et al., 2009; Ma et al…”

“However, the real world is not so simple: translating observers regularly undergo movements of their eyes and head (Royden et al., 1992, 1994; Banks et al., 1996; Crowell et al., 1998), and may also experience motion of other objects in the scene (Gibson, 1954; Warren & Saunders, 1995; Royden &…”

“Somatosensory and proprioceptive cues, in addition to efference copy signals, are especially critical for the guidance of active movements (Mittelstaedt & Mittelstaedt, 2001; Sun et al., 2004; Angelaki & Cullen, 2008) which comprise most natural self-motion.”

“show some degree of compensation for distortions of the flow field due to real or simulated eye movements (Bradley et al., 1996; Page & Duffy, 1999; Zhang et al., 2004; Bremmer et al., 2009), and they have”

“…by the work of Ma et al. (2006, 2008) is that cue reliability, perhaps in the form of a probability distribution over stimuli, is encoded implicitly by the activity of sensory neuronal populations (Foldiak, 1993; Sanger, 1996; Deneve et al., 2001; Knill & Pouget, 2004; Ma et al., 2006, 2008).”

“(Harris et al., 2000) used a match-to-sample task, asking subjects to estimate their distance traveled.”

“However, it remains unclear how these changes in weights (Morgan et al., 2008) might be related to our preliminary findings of a single-cell correlate of behavioral cue reweighting (measured with ROC-based neurometric functions, as discussed above).”

“priors favoring zero linear acceleration and angular velocity (MacNeilage et al., 2007; see also Laurens & Droulez, 2007).”

“To quantify neuronal sensitivity in a manner similar to behavior, we used signal detection theory (receiver operating characteristic (ROC) analysis; Green & Swets, 1966; Bradley et al., 1987; Britten et al., 1992) to generate neurometric functions for each cell in the three stimulus conditions.”

“However, other limitations of the vestibular system, such as its failure to encode sustained constantvelocity motion (Fernandez & Goldberg, 1976a; Büttner & Waespe, 1981; Si et al., 1997), suggest that the nervous system cannot rely solely on vestibular signals for heading perception, just as it…”

“Unlike otolith afferents, but similar to recently reported neurons in the cerebellar nodulus and uvula (Yakusheva et al., 2007), MSTd cells respond selectively to translation and not tilt relative to gravity (Liu & Angelaki, 2009).”

“studies of cue integration (e.g., Landy et al., 1995; Jacobs, 1999; van Beers et al., 1999; Landy & Kojima, 2001; Ernst & Banks, 2002; van Beers et al., 2002; Knill & Saunders, 2003; Oruc et al., 2003; Alais & Burr, 2004; Hillis et al., 2004; Jürgens & Becker, 2006; MacNeilage et al., 2007; Burr et al., 2009; Ma et al…”

“…theoretical framework, there remains substantial controversy in the literature as to which stimulus parameters are necessary and ⁄ or sufficient to produce vection (e.g., Zacharias & Young, 1981; Probst et al., 1985; Howard & Heckmann, 1989; Telford & Frost, 1993; Tarita-Nistor et al., 2006).”

“…translating observers regularly undergo movements of their eyes and head (Royden et al., 1992, 1994; Banks et al., 1996; Crowell et al., 1998), and may also experience motion of other objects in the scene (Gibson, 1954; Warren & Saunders, 1995; Royden & Hildreth, 1996; Warren & Rushton, 2009).”

“…& Fredrickson, 1971; Grüsser et al., 1990; Fukushima, 1997) are largely unresponsive to the types of optic flow stimuli used in the behavioral experiments described above (Chen et al., 2010), and thus seem unlikely to play a substantial role in visual–vestibular integration for heading perception.”

“…translating observers regularly undergo movements of their eyes and head (Royden et al., 1992, 1994; Banks et al., 1996; Crowell et al., 1998), and may also experience motion of other objects in the scene (Gibson, 1954; Warren & Saunders, 1995; Royden & Hildreth, 1996; Warren & Rushton, 2009).”

“bilateral labyrinthectomy (Gu et al., 2007; Takahashi et al., 2007).”

“translating observers regularly undergo movements of their eyes and head (Royden et al., 1992, 1994; Banks et al., 1996; Crowell et al., 1998), and may also experience motion of other objects in the scene (Gibson, 1954; Warren & Saunders, 1995; Royden & Hildreth, 1996;”

“…proprioceptive and motor-related signals as early as one or two synapses from the periphery, in the brainstem vestibular nuclei (Henn et al., 1974; Robinson, 1977; Waespe & Henn, 1977; Daunton & Thomsen, 1979) and vestibulocerebellum (Waespe et al., 1981; Waespe & Henn, 1981; Markert et al.,…”

“…world is not so simple: translating observers regularly undergo movements of their eyes and head (Royden et al., 1992, 1994; Banks et al., 1996; Crowell et al., 1998), and may also experience motion of other objects in the scene (Gibson, 1954; Warren & Saunders, 1995; Royden & Hildreth, 1996;…”

“…Ltd European Journal of Neuroscience, 31, 1721–1729 MSTd and VIP neurons have large receptive fields and directional selectivity for optic flow patterns that simulate self-motion (Tanaka et al., 1986; Tanaka & Saito, 1989; Duffy & Wurtz, 1991, 1995; Schaafsma & Duysens, 1996; Bremmer et al., 2002).”

“…proprioceptive and motor-related signals as early as one or two synapses from the periphery, in the brainstem vestibular nuclei (Henn et al., 1974; Robinson, 1977; Waespe & Henn, 1977; Daunton & Thomsen, 1979) and vestibulocerebellum (Waespe et al., 1981; Waespe & Henn, 1981; Markert et al., 1988).”

“…recognized as ‘vestibular cortex’ (e.g., area 2v and parieto-insular vestibular cortex; Schwarz & Fredrickson, 1971; Grüsser et al., 1990; Fukushima, 1997) are largely unresponsive to the types of optic flow stimuli used in the behavioral experiments described above (Chen et al.,…”

“…applied in a number of theoretical and psychophysical studies of cue integration (e.g., Landy et al., 1995; Jacobs, 1999; van Beers et al., 1999; Landy & Kojima, 2001; Ernst & Banks, 2002; van Beers et al., 2002; Knill & Saunders, 2003; Oruc et al., 2003; Alais & Burr, 2004; Hillis et al.,…”

“at least 50-60% of MSTd neurons are also selective for physical translation in darkness (Duffy, 1998; Bremmer et al., 1999; Gu et al., 2006).”

“show some degree of compensation for distortions of the flow field due to real or simulated eye movements (Bradley et al., 1996; Page & Duffy, 1999; Zhang et al., 2004; Bremmer et al., 2009), and they have”

“…translating observers regularly undergo movements of their eyes and head (Royden et al., 1992, 1994; Banks et al., 1996; Crowell et al., 1998), and may also experience motion of other objects in the scene (Gibson, 1954; Warren & Saunders, 1995; Royden & Hildreth, 1996; Warren & Rushton, 2009).”

“Results showed that physical motion (e.g., vestibular) cues dominated the estimation of self-motion, contrary to the findings of Ohmi (1996).”

“A recent series of experiments employing the ‘subjective visual vertical’ task (De Vrijer et al., 2008, 2009; Vingerhoets et al., 2009) extended these findings and supported the existence of an upright prior.”

“These and other modern computational methods for modeling spatial orientation are discussed further in a comprehensive review by MacNeilage et al. (2008).”

“The overarching hypothesis exemplified by the work of Ma et al. (2006, 2008) is that cue reliability, perhaps in the form of a probability distribution over stimuli, is encoded implicitly by the activity of sensory neuronal populations (Foldiak, 1993; Sanger, 1996; Deneve et al., 2001; Knill &…”

“(De Vrijer et al., 2008, 2009; Vingerhoets et al., 2009) extended these findings and supported the existence of an upright prior.”

“In addition, they show some degree of compensation for distortions of the flow field due to real or simulated eye movements (Bradley et al., 1996; Page & Duffy, 1999; Zhang et al., 2004; Bremmer et al., 2009), and they have been causally linked to heading perception from optic flow in…”

“…& Kojima, 2001; Ernst & Banks, 2002; van Beers et al., 2002; Knill & Saunders, 2003; Oruc et al., 2003; Alais & Burr, 2004; Hillis et al., 2004; Jürgens & Becker, 2006; MacNeilage et al., 2007; Burr et al., 2009; Ma et al., 2009), each grounded in mathematical principles borrowed from ideal…”

“…the neural underpinnings of not only heading perception but also classical phenomena such as vection, the somatogravic and oculogravic illusions (Clark & Graybiel, 1963; Wolfe & Cramer, 1970; Previc et al., 1992), and biases in subjective verticality (Howard, 1982; De Vrijer et al., 2009).”

“…by the work of Ma et al. (2006, 2008) is that cue reliability, perhaps in the form of a probability distribution over stimuli, is encoded implicitly by the activity of sensory neuronal populations (Foldiak, 1993; Sanger, 1996; Deneve et al., 2001; Knill & Pouget, 2004; Ma et al., 2006, 2008).”

“…theoretical framework, there remains substantial controversy in the literature as to which stimulus parameters are necessary and ⁄ or sufficient to produce vection (e.g., Zacharias & Young, 1981; Probst et al., 1985; Howard & Heckmann, 1989; Telford & Frost, 1993; Tarita-Nistor et al., 2006).”

“Results showed that physical motion (e.g., vestibular) cues dominated the estimation of self-motion, contrary to the findings of Ohmi (1996).”

“Somatosensory and proprioceptive cues, in addition to efference copy signals, are especially critical for the guidance of active movements (Mittelstaedt & Mittelstaedt, 2001; Sun et al., 2004; Angelaki & Cullen, 2008) which comprise most natural self-motion.”

“Although the notion of frequencybased weighting has been disputed (Probst et al., 1985), future studies could attempt to reconcile these findings by explicitly acknowledging the probabilistic nature of the task and by seeking to measure and manipulate quantities such as cue reliability or the prior…”

“Journal Compilation ª Federation of European Neuroscience Societies and Blackwell Publishing Ltd the semicircular canals (reviewed by Angelaki & Cullen, 2008; Angelaki & Yakusheva, 2009).”

“However, the weights were determined not by cue reliability per se but by factors such as the frequency content of the stimuli and the magnitude of the conflict between the two cues (Zacharias & Young, 1981; see also Wright et al., 2005).”

“…of the vestibular system, such as its failure to encode sustained constantvelocity motion (Fernandez & Goldberg, 1976a; Büttner & Waespe, 1981; Si et al., 1997), suggest that the nervous system cannot rely solely on vestibular signals for heading perception, just as it cannot rely solely on…”

“Similarly, areas traditionally recognized as ‘vestibular cortex’ (e.g., area 2v and parieto-insular vestibular cortex; Schwarz & Fredrickson, 1971; Grüsser et al., 1990; Fukushima, 1997) are largely unresponsive to the types of optic flow stimuli used in the behavioral experiments described above…”

“Furthermore, the temporal dynamics of vestibular responses in MSTd are generally consistent with the encoding of stimulus velocity (Gu et al., 2006; Liu & Angelaki, 2009), similar to the dynamics of visual motion responses.”

“…proprioceptive and motor-related signals as early as one or two synapses from the periphery, in the brainstem vestibular nuclei (Henn et al., 1974; Robinson, 1977; Waespe & Henn, 1977; Daunton & Thomsen, 1979) and vestibulocerebellum (Waespe et al., 1981; Waespe & Henn, 1981; Markert et al., 1988).”

“This lack of visual–vestibular convergence was also observed in vestibular-responsive areas of the thalamus (Meng & Angelaki, 2010).”

“…theoretical framework, there remains substantial controversy in the literature as to which stimulus parameters are necessary and ⁄ or sufficient to produce vection (e.g., Zacharias & Young, 1981; Probst et al., 1985; Howard & Heckmann, 1989; Telford & Frost, 1993; Tarita-Nistor et al., 2006).”

“In fact, a recent study (Bryan & Angelaki, 2009) showed relatively weak convergence of visual and vestibular inputs in the vestibular and fastigial nuclei when animals were required to fixate a head-fixed target (suppressing OKN).”

“neurons, an important follow-up question is whether and how this representation is decoded into a more explicit form (e.g., Jazayeri & Movshon, 2006; Jazayeri, 2008) that can be used to guide decisions”

“, 1974; Robinson, 1977; Waespe & Henn, 1977; Daunton & Thomsen, 1979) and vestibulocerebellum (Waespe et al., 1981; Waespe & Henn, 1981; Markert et al., 1988).”

“The behavioral and perceptual benefits of multisensory integration have been shown to be quite impressive and to confer a highly degree of adaptive benefit (Calvert et al., 2004; Fetsch et al., 2010; Murray and Wallace, 2012; Stein and Meredith, 1993; Stevenson et al., 2014a).”

“, vision (Fetsch et al., 2010), audition, somatosensation (Lopez et al.”

“, the visual-inertial SFM problem (from now on the Vi-SfM problem), has particular interest and has been investigated by many disciplines, both in the framework of computer science Bryson and Sukkarieh (2008), Jones and Soatto (2011), Kelly and Sukhatme (2011), Martinelli (2012), Strelow and Singh (2004) and in the fr…”

“Several results suggest that a process based on dynamic sensory reweighting may be important to explain cybersickness and adaptation to virtual environments (Butler et al. 2010; de Winkel et al. 2010; Fetsch et al. 2010; Angelaki et al. 2011).”

“Previous studies have suggested a role for MSTd in multisensory heading perception (for reviews, see Angelaki et al., 2009; 2011; Fetsch et al., 2010).”

“However, the most thoroughly studied area is the dorsal medial superior temporal area (MSTd) (Britten, 2008; Angelaki et al., 2009; Fetsch et al., 2010).”

“Near-optimalBayesian integration of visual and vestibular information has been measured in several studies (Gu et al. 2008; Butler et al. 2010; Fetsch et al. 2009; Prsa et al. 2012; Drugowitsch et al. 2014; Fetsch et al. 2010).”

“Neural selectivity for heading has been described extensively in the dorsal medial superior temporal (MSTd) area, where many neurons show heading tuning in response to both inertial motion and optic flow (Duffy, 1998; Bremmer et al., 1999; Gu et al., 2006; Britten, 2008; Fetsch et al., 2010; Angelaki et al., 2011).”

“…neurons in MSTd respond during vestibular stimulation in the dark and can display the same or opposite tuning for direction of motion in both visual and vestibular modalities, suggesting multimodal interactions in encoding heading (Bremmer et al., 1999; Gu et al., 2006; Fetsch et al., 2007, 2010).”

“The Bayesian framework has proven extremely successful in accounting for the behavioral (Fetsch et al., 2010) and single-cell recording (Gu et al.”

“, the weight of each cue is proportional to its reliability (1/variance) (171).”

“…some earlier threshold studies have investigated optimal integration by degrading the precision of one or more of the cues (Ernst and Banks 2002; Fetsch et al. 2010), we chose to vary frequency as for our earlier threshold study that investigated optimal integration of visual and vestibular…”

“Therefore, vestibular influences on somatosensation do not seem to act as an additional informative input contributing to multisensory integration (Fetsch et al., 2010).”

“…2010; Beierholm et al., 2009; Besson et al., 2010; Charbonneau et al., 2013; Clemens et al., 2011; Deneve and Pouget, 2004; Ernst and Banks, 2002; Fetsch et al., 2010, 2012, 2009; Helbig et al., 2012; Ma and Pouget, 2008; Nath and Beauchamp, 2011; Noppeney et al., 2010; Rohe and Noppeney, 2015).”

“…numerosity judgment (Shams et al., 2005; Bresciani et al., 2006; Wozny et al., 2008), rate perception (Roach et al., 2006), oddity perception (Fetsch et al., 2009, 2010; Butler et al., 2010), angular displacement (Jürgens and Becker, 2006), gravitoinertial force discrimination (Macneilage et…”

“…by simultaneous sounds (Alais and Burr, 2004; Beer and Roder, 2004; Lewis and Noppeney, 2010; Schmiedchen et al., 2012) or vestibular information (Fetsch et al., 2010; Gu et al., 2008), even so when the multisensory stimulus is not directly task relevant (Gleiss and Kayser, 2014b; Kim et al.,…”

“Visuo-vestibular cue integration is essential for self-motion, motor control and spatial orientation and hence for the interaction with our environment (for a review see Fetsch et al., 2010) and the demarcation of self from non-self (Lopez, 2013).”

“(Duffy 1998; Gu et al. 2006; Fetsch et al. 2010), a recent inactivation study in macaques provided causal evidence that MSTd is dominant for visual heading direction thresholds; VPS and parieto-insular vestibular cortex (PIVC) areas are dominant for vestibular signals, but inactivation of VPS had no functional consequ…”

“…flow and vestibular signals related to self-translation occurs in both the dorsal medial superior temporal area (MSTd; Gu et al., 2006, 2008, 2012; Fetsch et al., 2007, 2009, 2010, 2012; Chen et al., 2008; Yang et al., 2011) and the ventral intraparietal area (VIP; Bremmer et al., 2002a,b, 2005,…”

“The cortical area currently considered the most likely site of convergence of visual-vestibular heading information is the dorsal medial superior temporal area (MSTd) [6, 7], as neurons at this site have been shown to respond to optic flow patterns suggesting self-motion [8–14] as well as to translation in absence of…”

“…i.e., the visual-inertial structure from motion problem (from now on the Vi-SfM problem), has particular interest and has been investigated by many disciplines, both in the framework of computer science [2], [12], [13], [20], [23] and in the framework of neuroscience (e.g., [1], [5], [8]).”

“Most of the studies that have addressed walking in blindfolded subjects, however, have been focused on pathintegration or orientation towards a visual target presented prior to initiating movement, or reproduction of a recently performed route [33,34,35,36].”

“…integration has been studied by providing motion and other sensory cues that are either congruent or conflicting and determining how much weight the brain places on each cue to form an overall estimate of head orientation or movement in the environment (Fetsch et al. 2010; McCollum et al. 1996).”

“…levels (e.g., Alais & Burr, 2004; Battaglia et al., 2003; Butler et al., 2010; Butler et al., 2011; Camors et al., 2015; Ernst & Banks, 2002; Girshick & Banks, 2009; Hillis et al., 2004; Jacobs, 1999; Koppen et al., 2019) and neural levels (e.g., Fetsch et al., 2010, 2013) during perceptual tasks.”

“This involves monitoring sensory flow (mostly from proprioception, vision, and audition) to update rotation and translation of the body relative to the starting point [17].”

“These novel findings extend previous demonstrations that vestibular cues pertaining to self-motion perception are processed according to statistical optimality (Butler et al. 2010; Fetsch et al. 2010; Laurens and Droulez 2007; MacNeilage et al. 2008; Prsa et al. 2012; Vingerhoets et al. 2009).”

“…a number of studies suggest that the brain optimally processes this information (Borah et al. 1988; Butler et al. 2010; de Winkel et al. 2010; Fetsch et al. 2009, 2010, 2011; Gu et al. 2008; Jurgens and Becker 2006; Karmali et al. 2014; Karmali and Merfeld 2012; Laurens and Droulez 2007;…”

“…involves a complex process that relies on the optimal integration of information from the visual, vestibular, proprioceptive, and auditory modalities (Angelaki and Cullen 2008; Chang, Uchanski, and Hullar 2012; Fetsch et al. 2009; Fetsch, DeAngelis, and Angelaki 2010; Lord and Sturnieks 2005).”

“Much of the research that is currently available which addresses how sensory cues are integrated to inform heading perception has been conducted in primates (Gu et al., 2007; Morgan et al., 2008; Angelaki et al., 2009; Fetsch et al., 2010; Chen et al., 2011; Cullen, 2012; Fetsch et al., 2013; Smith et al., 2017).”

“In order to optimally combine different information from across these sensory systems, the reliability of the inputs must be ascertained (Cheng et al., 2007; Deneve and Pouget, 2004; Fetsch et al., 2010).”

“…et al., 2014;Wystrach et al., 2015; Fleischmann et al., 2016) and evidence from landmark navigation studies in pigeons (Sutton, 2002; Blaisdell and Cook, 2005; Cheng et al., 2006) or cue integration inmonkeys (Fetsch et al., 2010; Dokka et al., 2015) also indicates close to optimal cue integration.”

“This problem has been investigated by many disciplines, both in the framework of computer science [7], [22], [23], [29], [34] and in the framework of neuroscience (e.g., [5], [10], [11]).”

“These heading thresholds are comparable to those found for both macaques and humans in previous studies (Gu et al., 2007, 2008; Fetsch et al., 2010; Chen et al., 2013).”

“[51] Fetsch CR, DeAngelis GC, Angelaki DE.”

“For vertebrates, several senses are available to detect self-motion, notably vestibular and visual systems (1).”

“…this correspondence, which is in line with previously reported feature fusion mechanisms observed within and between visual, somato-sensory, vestibular, and auditory sensory modalities (Treisman and Gelade, 1980; Angelaki et al., 2009; Fetsch et al., 2010; Parise et al., 2012; Ernst et al., 2013).”

“This also stands in a visuo-vestibular heading discrimination task [20,64], which is embodied by steeper psychometric curve in the combined condition compared with that in either single-cue condition, as predicted by equation (3.4).”

“When cues are available, combining them facilitates the detection of salient events and reduces perceptual uncertainty, in order to improve reactions to immediate dangers and to rapidly varying events [2].”

“In this issue Fetsch, DeAngelis and Angelaki describe how Bayesian decision theory accounts for the contributions of visual and vestibular information to estimations of one’s translation in space (Fetsch et al., 2010).”

“When visual and vestibular cues are congruent and redundant (as is typically the case during most everyday experiences), integrating these inputs improves the precision of perceptual estimates (Ernst and Bülthoff, 2004; Fetsch et al., 2010; Butler et al., 2015).”

“Particularly, it has been largely explained by the theory of optimal integration (Fetsch et al, 2010, 2013; ter Horst et al., 2015).”

“These multiple sensory cues are integrated to reduce HD errors (Telford et al., 1995; Becker et al., 2002; Fetsch et al., 2010; Frissen et al., 2011).”

“The visual-inertial structure from motion problem (from now on the Vi-SfM problem), has particular interest and has been investigated by many disciplines, both in the framework of computer science [3], [10], [11], [15], [17], [21] and in the framework of neuroscience (e.g., [2], [4], [5]).”

“In such situations, the visual and inertial stimuli are integrated near optimally based on their relative reliabilities or Bayesian ideal (Knill and Pouget 2004; Fetsch et al. 2010; Crane 2017).”

“This filtering-out of a degraded signal has been shown during visuo-vestibular integration for monkey heading perception ( Fetsch et al., 2010 ), or for human object discrimination using visual and haptic senses ( Ernst and Banks, 2002 ).”

“Verticality perception is crucial for postural stability, spatial awareness, and precise motor coordination (Lacquaniti et al. 2014; White et al. 2020; Glasauer and Mittelstaedt 1998; Fetsch et al. 2010; Kheradmand and Winnick 2017).”

“We found that intra-auditory integration enhanced both online and offline motor synergy.”

“Since IN involves a physical motion and CE does not, one could raise the point that PSE measurements are actually measuring visual-vestibular integration, and not vection which is purely visual [42,43].”

“The third hypothesis states that perceived motion is based on an optimal weighted average of visual and vestibular sensory inputs (Fetsch et al., 2010).”

“According to sensory reweighting (Asslander and Peterka 2014 ; Peterka 2002 ) theory, and an optimal cue integration framework, the nervous system down-weights unreliable sensory channels and up-weights the most pre - cise channels (Ernst and Banks 2002 ; Fetsch et al. 2010 ).”