Visual and Nonvisual Contributions to Three-Dimensional Heading Selectivity in the Medial Superior Temporal Area

“It can be shown that such a network is capable of estimating an arbitrary function, given a sufficient number of hidden units (Bishop, 1995).”

“The overall heading estimate of the network, resulting from the activity of the output units, was defined by a population vector (Georgopoulos et al., 1986) and was computed as the vector sum of the responses of the 26 output units.”

“Both monkeys were also implanted with scleral coils for measuring eye movements in a magnetic field (Robinson, 1963).”

“As demonstrated previously by Zipser and Andersen (1988), the transformation of optic flow signals from an eye-centered to head-centered reference frame was implemented in the hidden layer of our model through modulation of hidden unit responses by eye position (i.”

“First, reference frames for visual and vestibular heading signals in MSTd remain distinct, although evidence clearly shows that these neurons might mediate multisensory cue integration (Gu et al. 2006b).”

“Details of the apparatus and stimuli (Gu et al., 2006), as well as the basic task design and training (Gu et al.”

“…medial superior temporal cortex of monkeys, which detects selfmotion from vestibular and visual inputs, and distinguishes them from object motion and updates spatial orientation (Geesaman and Andersen, 1996; Duffy, 1998; Gu et al., 2006; Fetsch et al., 2007; see Shinder and Taube, 2010 for review).”

“Furthermore, several other cortical areas in the monkey have an identified homologue in the human (e.g., MT/MST, Anterior Intraparietal Area (AIP)).”

“In area MSTd, many neurons have heading tuning that is not matched for visual and vestibular inputs 13 .”

“The hypothesis that MSTd contributes to the non-visual perception of heading is supported by recent findings that MSTd neurons show directional tuning in the absence of visual stimulation, presumably reflecting vestibular input”

“conversion of heading signals into a head-centered reference (88) than the generation of a robust heading representation (218).”

“Finally, MST seems to integrate signals from the vestibular and visual modalities [48], thus it is considered to be a multisensory area.”

“Consistent with previous findings (Gu et al., 2006; Takahashi et al., 2007), many MSTd neurons were tuned to heading direction, and their responses mainly followed the Gaussian velocity profile of the stimulus (Figure 1A).”

“First, what is the combination rule used by neurons to integrate sensory signals from two different sources? Second, how does this rule depend on the relative reliabilities of the sensory cues? We addressed these issues by examining visual-vestibular interactions in the dorsal portion of the medial superior temporal a…”

“In response to translation, both congruent and opposite neurons were encountered in VIP with roughly equal frequencies, similar to MSTd (Gu et al., 2006, 2010).”

“…of area MST (MSTd) respond preferentially to specific optic flow components (expansion, contraction, and rotation; Duffy and Wurtz 1991b) and are sensitive to the direction of heading in the visual and vestibular domains (Duffy and Wurtz 1995; Page and Duffy 2003; Gu et al. 2006, 2007, 2008).”

“Thus, for each stimulus direction and motion type, we computed the firing rate during the middle 1 s interval of each trial and averaged across stimulus repetitions to compute the mean firing rate (Gu et al., 2006b).”

“…in comparison with the extensive visual/vestibular convergence seen in extrastriate areas, e.g., the dorsal medial superior temporal area (MSTd) (Bremmer et al., 1999; Page and Duffy, 2003; Gu et al., 2006; Takahashi et al., 2007) and the ventral intraparietal area (VIP) (Schlack et al., 2002).”

“As reported previously (Gu et al., 2006), the distribution of heading preferences (Figures 4B and 4D) is bimodal for both the vestibular and visual conditions, with peaks at 90 and 90 azimuth (lateral headings).”

“Furthermore, the temporal dynamics of vestibular responses in MSTd are generally consistent with the encoding of stimulus velocity (Gu et al., 2006; Liu & Angelaki, 2009), similar to the dynamics of visual motion responses.”

“3c).”

“This finding contrasts with previous results from area MSTd, for which combined responses under identical stimulus conditions (100% motion coherence) were generally dominated by the visual tuning (Gu et al., 2006; Takahashi et al., 2007; Morgan et al., 2008).”

“…different cortical areas interact and how computations within each area relate to an overall vestibular sensation of self-motion, although some progress has been made for certain regions [e.g., the medial superior temporal area (MST); see Bremmer et al. 1999; Duffy 1998; Gu et al. 2006, 2008].”

“This information will yield a basic understanding of thalamic neural processing of vestibular signals, and provide insight into previous advances regarding cortical use of vestibular information (Fukushima et al., 2000, 2004; Chen et al., 2006, 2007; Gu et al., 2006, 2007).”

“Most recordings concentrated on the posterior/medial portions of MSTd, corresponding to more eccentric, lower hemifield receptive fields in the underlying area MT (Gu et al., 2006, 2007).”

“This may appear to conflict with data from multimodal areas MSTd, VIP, and VPS, where neural activity in response to self-motion reflects a mixture of velocity and acceleration components (Fetsch et al., 2010; Chen et al., 2011a).”

“Many macaque MSTd neurons also respond to vestibular stimulation arising from actual forward motion (Bremmer et al. 1999; Gu et al. 2006; Fetsch et al. 2007), and there is evidence that these signals feed into heading perception (Gu et al. 2007, 2008).”

“Both areas show selectivity for optic flow patterns that simulate self-motion, as well as directional tuning for inertial motion in darkness (MSTd; Duffy and Wurtz, 1995; Lappe et al., 1996; Duffy, 1998; Page and Duffy, 2003; Gu et al., 2006) (VIP; Bremmer et al.”

“%) neurons are tuned to inertial motion than to optic flow (Duffy, 1998; Page and Duffy, 2003; Gu et al., 2006; Takahashi et al., 2007); (2) even for signif- icantly tuned neurons, the strength of tuning is weaker for vestibular than visual stimuli (Duffy, 1998; Page and Duffy, 2003; Gu et al.,…”

“…suggests that humans and monkeys can combine visual and vestibular cues to improve self-motion perception (Telford et al., 1995; Ohmi, 1996; Harris et al., 2000; Bertin and Berthoz, 2004; Gu et al., 2006b), but it remains unclear exactly where and how the brain carries out this sensory integration.”

“Following up on earlier observations using verbal reports (Denise et al., 1988; Guedry, 1974), Vingerhoets and colleagues (2006, 2007) used a 2AFC task to quantify these sensations in human subjects.”

“…to expand and facilitate the study of the integration between vision and proprioception (Van Beers et al., 1999) or between visual and vestibular stimuli (Gu et al., 2006), as well as extend the study of the body schema, which is usually studied for hand interactions (Maravita et al., 2003).”

“Although some of these data have been included in previous publications (Gu et al., 2006; Takahashi et Received Aug. 26, 2010; revised Oct. 18, 2010; accepted Dec. 28, 2010.”

“Other studies have provided a better understanding of the brain areas involved in the integration of visual motion signals with non-visual cues for the perception of selfmotion (Angelaki and Hess 2005; Bremmer et al. 2002; Fetsch et al. 2007; Gu et al. 2007; Gu et al. 2006; Muller et al. 2005; Page and Duffy 2003).”

“stabilization as described previously (Gu et al., 2006), as well as two scleral”

“MSTd/VIP neurons are also selective for motion in darkness, suggesting that they receive vestibular inputs (Duffy, 1998; Bremmer et al. 1999, 2002b; Schlack et al. 2002; Page & Duffy, 2003; Gu et al. 2006, 2007, 2008; Takahashi et al. 2007).”

“Just over half of the MST neurons that respond to optic flow stimulation also also respond to head motion sensitivity [ 31, 69, 97].”

“To reconstruct the locations of injection sites, we analyzed MRI images as follows (for details, see Gu et al., 2006; Chen et al., 2010).”

“…of the human MT+ region that responds well to optic-flow stimuli, showing that a portion of neurons also receive extraretinal (vestibular: Duffy 1998; Gu et al. 2006; smooth pursuit eye-movements: Ono and Mustari 2006; command related corollary discharges: Sommer and Wurtz 2004) inputs as well.”

“…neurons in MSTd respond during vestibular stimulation in the dark and can display the same or opposite tuning for direction of motion in both visual and vestibular modalities, suggesting multimodal interactions in encoding heading (Bremmer et al., 1999; Gu et al., 2006; Fetsch et al., 2007, 2010).”

“In contrast, neurons in the dorsal medial superior temporal area (MSTd) of visual cortex appear to respond with a lateral bias to passive translations (Gu et al., 2006; Takahashi et al., 2007).”

“In fact MSTd neurons can take optic flow and compute the heading or direction of self-motion (Duffy and Wurtz, 1995; Gu et al., 2006).”

“work: in primates (Bremmer et al. 1999; Gu et al. 2006) and in humans (Smith et al.”

“Visual–vestibular convergence has also been reported in the extrastriate visual area MST, a major region for self-motion perception based on optic flows (Bremmer et al., 1999; Gu et al., 2006).”

“Cerebral Cortex April 2012;22:865--876 doi:10.1093/cercor/bhr154 Advance Access publication June 27, 2011”

“…of vestibular – visual integration have identified two types of multisensory neurons with similar prevalence in MSTd and VIP; those that are tuned to vestibular and visual motion in the same direction (congruent neurons) and those that prefer opposite directions (opposite neurons) (6, 8, 11).”

“Responses were computed as mean firing rates over the middle 1s period of the stimulus 35 , or within at least five consecutive 400 ms bins, 25 ms apart.”

“Particularly relevant might be a group of corticalmultisensory neuronswith incongruent visual and vestibular preferences [26,28,39].”

“…brain regions of the 49 monkey, such as the dorsal medial superior temporal area (MSTd, 50 Duffy and Wurtz, 1991; Froehler and Duffy, 2002; Gu et al., 2006; 51 Lagae et al., 1994), ventral intra-parietal area (VIP, Chen et al., 52 2011), parieto-occipital V6 area (Galletti et al., 2001),…”

“Platform movements and visual stimuli were controlled by computer at 60 Hz, and the measured transfer function of the system (verified by accelerometer measurements) allowed us to accurately record the platform's position at all times (Gu et al., 2006, 2007).”

“Thus, overall, FEFsem neurons showed robust heading selectivity thatwas comparable to, but not better than, the selectivity of MSTd and VIP neurons.”

“The work reported here focussed on area hMST (human MST) because MSTd is a key area for visual–vestibular interaction in macaques (Fetsch et al., 2013; Gu et al., 2006; Takahashi et al., 2007).”

“…in the cortex, such as parieto-insular vestibular cortex (Guldin and Grüsser, 1998; Chen et al., 2010; Liu et al., 2011), medial superior temporal area (Gu et al., 2006, 2007; Fetsch et al., 2010; Chen et al., 2011a), and ventral intraparietal area (Schlack et al., 2002; Chen et al., 2011a,b).”

“Many MSTd and VIP neurons also receive vestibular input (Duffy 1998; Gu et al. 2006; Chen et al. 2011b; Fetsch et al. 2012), consistent with a role in encoding selfmotion.”

“Although neuronal responses to multisensory self-motion stimuli in parietal areas such as the medial superior temporal (MSTd) and ventral intraparietal (VIP) areas are well documented (Britten and van Wezel 1998, 2002; Bremmer, Duhamel, et al. 2002; Bremmer, Klam, et al. 2002; Chen et al. 2011; Gu et al. 2006; Macioka…”

“The first evidence of neurons encoding expansion and contraction was reported in MSTd (Saito et al.”

“Cortical area MST of the macaque monkey has been implied in self-motion processing based on its selectivity to optic Xow (Tanaka and Saito 1989; DuVy and Wurtz 1991a; Lagae et al. 1994) and vestibular self-motion cues (Bremmer et al. 1999; Froehler and DuVy 2002; Gu et al. 2006, 2008 ), and because microstimulation in…”

“For each stimulus direction and motion type, we computed neural firing rate during the middle l s interval of each trial and averaged across stimulus repetitions to compute the mean firing rate (Gu et al., 2006).”

“In addition, neurons in area MST, a major input source to area VIP, are modulated by vestibular stimulation (Ilg et al. 2004; Gu et al. 2006).”

“The yaw motor, a rear-projection screen, and a video projector were all secured onto a sixdegree-of-freedom motion platform that allowed physical translation (1).”

“Within MSTd not all headings are represented equally [47,48], and recent modeling using a population vector decoder [30,32,49] has demonstrated that this can explain why both visual and vestibular heading discrimination becomes worse with more eccentric trajectories [8].”

“Importantly, these multisensory MSTd neurons come in two types: congruent cells and opposite cells (Gu et al., 2006, 2008).”

“The model weights can be learned by the agent through biologically plausible self-supervision, since the approximate parameters of self-motion are known to the agent, via corollary discharges and vestibular and proprioceptive inputs [28].”

“Many MSTd and VIP neurons also receive vestibular input (Duffy et al. 1976; Gu et al. 2006; Chen et al. 2011a) and there is evidence that visual and vestibular cues are efficiently integrated by such neurons, with weightings based on cue reliability (Fetsch et al.”

“MSTd neurons are also tuned during actual self-motion (Duffy, 1998; Bremmer et al., 1999; Page and Duffy, 2003; Gu et al., 2006; Fetsch et al., 2007) and this selectivity is of vestibular origin (Takahashi et al., 2007).”

“(Duffy 1998; Gu et al. 2006; Fetsch et al. 2010), a recent inactivation study in macaques provided causal evidence that MSTd is dominant for visual heading direction thresholds; VPS and parieto-insular vestibular cortex (PIVC) areas are dominant for vestibular signals, but inactivation of VPS had no functional consequ…”

“Visual stimuli were programmed using the OpenGL graphics library and generated using an OpenGL accelerator board (Quadro FX 3000G, PNY Technologies; for details, see Gu et al. (2006)).”

“At each module, there exist two groups of excitatory neurons: congruent and opposite neurons (blue and red circles in Figure 4A respectively), and they have the same number of neurons, as supported by experiments (Figure 2C) (Chen et al., 2011; Gu et al., 2006).”

“This area receives anatomical input from MSTd and VIP (Boussaoud et al. 1990), two areas heavily implicated in the coding of self-motion (Chen et al. 2011, 2013; Gu et al. 2006, 2008).”

“In the past decade, this suggestion has been confirmed and elaborated; strong evidence now exists for visual–vestibular integration in macaque MSTd. Gu et al. (2006) examined visual and vestibular heading tuning in MSTd neurons in detail.”