Learning Invariance from Transformation Sequences

“, 2000), or more directly by a learning rule that superimposes past stimuli (Edelman & Weinshall, 1991; Földiak, 1991; Wallis & Bülthoff, 1999).”

“A great number of approaches [ 3 ,13,10,5] have been taken for robustness in pattern recognition.”

“Another approach to natural images is based on temporal coherence or stability (Földiák, 1991; Hashimoto, 2003; Hurri and Hyvärinen, 2003; Wiskott and Sejnowski, 2002).”

“A similar principle had been proposed for translation invariance (Földiák, 1991), but Rolls extended this to all types of invariance, and outlined how this could be set up in a hierarchical model (Rolls, 1992).”

“Such an idea is well-known in the theory of linear BSS. 44,67,68 The nonstationarity is supposed to be slow compared to the sampling rate, allowing us to consider the distributions inside each segment to be constant over time and resulting in a piece-wise stationary process.”

“Bypresentingconsistentsequencesoftrans-forming objects, the cells in the network can learn to respond to the same object through all of its naturally transformed states, as described by Földiák (1991), Rolls (1992, 2012a), Wallis et al. (1993), and Wallis and Rolls (1997).”

“…sought to explain how transform-invariant representations could arise through unsupervised training with real-world visual input are trace learning (Földiák, 1991; Rolls & Milward, 2000; Wallis & Rolls, 1997), and Continuous Transformation (CT) learning (Perry, Rolls, & Stringer, 2006; Stringer,…”

“This broad concept falls under the broad concept of slowness regularization (Hinton, 1989; Földiák, 1991; Luciw & Schmidhuber, 2012; Jonschkowski & Brock, 2015; Wen et al., 2015) which attempts to minimize L(f(xt), f(xt+1)) where L is a loss function describing the distance between f(xt) and…”

“…temporal learning models in computational neuroscience, the brain builds invariant object representations by extracting smoothly changing features from the visual environment (DiCarlo et al., 2012; Feldman & Tremoulet, 2006; Földiák, 1991; Rolls, 2012; Stone, 1996; Wiskott & Sejnowski, 2002).”

“The trace-learning rule (Fo¨ldia´k, 1991; Rolls, 1992; Wallis & Rolls, 1997; Rolls & Milward, 2000) encourages neurons to develop invariant responses to input patterns that tend to occur close together in time, because these are likely to be from the same object.”

“…transform invariant representations in a 1-layer competitive network through trace learning An approach to invariance learning, trace learning (Fo¨ldia´k 1991; Wallis and Rolls 1997; Rolls and Milward 2000; Rolls and Stringer 2001), relies on the temporal continuity of visual stimuli in the…”

“LETTER Communicated by Ralph M. Siegel”

“(Note that this differs from the global " winner-take-all " paradigm implemented by Földiák 1991.)”

“A. Related Research 1) SFA in Computer Vision: Inspired by the slowness principle of neural networks [7], [5] employs SFA for invariant localization and recognition.”

“This principle has probably first been formulated by Hinton (1989), and online learning rules were developed shortly after by Földiák (1991) and Mitchison (1991).”

“For learning general representations from video in an unsupervised manner, minimizing the variation across time is seen both in biological, e.g. Földiák (1991), Wiskott and Sejnowski (2002), and arti-ficial, e.g. Goroshin et al. (2015) systems.”

“According to the learning scheme used in earlier models ( Foldiak, 1991; Wallis, 1996), the strength of the training signal which a neuron receives, and hence the size of the change in appearance that it will be able to accommodate in one step, is dependent on the amount of time it has responded strongly to stimuli in…”

“invariant object representations have been proposed Foldiak (1991) ; O’Reilly and Johnson (1994), which”

“One of the proposed principles is known as temporal coherence or temporal smoothness [1,2,3,4].”

“, [63, 64, 65, 66, 67, 68]).”

“…(Swindale, 1996; von der Malsburg, 1973; F ¨ oldi ´ ak, 1989, 1990; Marshall, 1995; Sirosh and Miikkulainen, 1994; Oja, 1989; Sanger, 1989; O’Reilly, 1998; Rumelhart and Zipser, 1985; Kohonen, 1997; Grossberg, 1987; Ritter et al., 1992; Hertz et al., 1991; F ¨ oldi ´ ak, 1991; Wallis, 1996).”

“Using this approach, it has been shown that optimally sparse representations resembling so-called simple cells [48, 37] and optimally stable representations giving rise to complex cells [80, 18, 83] can be learned from visual stimulus consistent with physiology of the primary visual cortex V1.”

“This is also related to previous approaches that try to train neural networks via exploiting the spatial continuity of moving objects [14, 38, 8].”

“• A modified associative (Hebb-like) learning rule incorporating a temporal trace of each neuron’s previous activity, which, it has been shown (Földiák, 1991; Rolls, 1992; Wallis et al., 1993; Wallis and Rolls, 1997; Rolls and Milward, 2000; Rolls, 2012), enables the neurons to learn transform…”

“These are also scenarios under which temporal association is presumed to operate (Edelman and Weinshall 1991; Fo« ldiak 1991; Wallis and Bu« lthoff 1999).”

“For η = 0 the trace becomes just the present firing rate, and as η is increased the contribution of preceding firing at times earlier than the current timestep is increased (Földiák 1991; Stringer et al. 2000a, 2000b; Rolls & Deco 2002).”

“3 , 108, 132–134 ).”

“The terms of the horizontal products of both pathways can be presented as follows: xvk = ∑ j svj u v kj; xdk = ∑ l sdl u d kl (6) The output of the two streams composes a horizontal product for the network output as we defined in Equation (1).”

“As Földiàk suggested in [27], it makes sense to consider the learning equation as being dW dt = −κW + (v ∗ g1) ⊗ (v ∗ g1), where ∗ is the convolution and g1 : t ∈ R → β1e−β1tH(t).”

“Trace learning utilizes the temporal continuity of objects in the world to learn transformation-invariant representations (Földiák, 1991).”

“models of learning invariances from spatiotemporal dependencies (e.g. [10, 12, 28 , 56, 66, 77]).”

“The approach described in this paper is conceptually most closely related to work by Földiák (1991),Mitchison (1991), Becker & Hinton (1995) and Stone (1996).”

“The last degree of freedom is eliminated by demanding order (Equation 10), leading to the following constraints: The principle of slowness, although not the above definition, has been used very early in the context of neural networks (F¨oldi´ak, 1991; Becker and Hinton, 1992).”

“In [9], the author has proposed a novel learning method which allows the network to learn transformation invariance by introducing a modified Hebbian learning approach between the simple and complex units which is a good way to implement positional invariance criteria but still the method is not able to extract invari…”

“In our analysis, a maximum-likelihood decoder (Földiák, 1991; Sanger, 1996; Jazayeri and Movshon, 2006; Ma et al., 2006; Graf et al., 2011; Fetsch et al., 2011) of the stimulus was constructed based on the neural responses in the discrimination task and then applied to the data for categorization task to reconstruct t…”

“The learning process involves what is essentially a simple Hebbian associative learning rule, but which includes a short memory trace of preceding neuronal activity (in, for example, the post-synaptic term) [28,79,106,110,133].”

“The general idea of imposing a ‘slowness prior’ was initially proposed by [20] and [59] to extract invariant or slowly varying features from temporal sequences rather than using static orderless data points.”

“CT learning is compared in Experiment 2 with another approach to invariance learning, trace learning, and we summarize next the trace learning procedure developed and analyzed previously (Fo¨ldia´k 1991; Rolls 1992; Wallis and Rolls 1997; Rolls and Milward 2000; Rolls and Stringer 2001).”

“Using the concept of the trace rule, where the response is determined by the current and past responses [38], the average of N sets of simple cell responses is then used as the input to the complex cells; i.”

“Other works [1, 14, 38, 42, 52, 45] apply consistency constraints to predictions from the same input with different transformations in order to obtain perturbation-invariant representations.”

“This principle has probably first been formulated by Hinton [13] and online learning rules were developed shortly after [9, 22].”

“In order to relax these constraints, we have recently demonstrated that finding a policy that maximizes the total response across the entire population of disparity neurons used to represent disparity leads to the autonomous development of vergence control to maintain fixation [29].”

“) and a complex cell may develop is to invoke a Hebbian trace rule ([35], see also [10], [89]).”

“Various types of self-organization rules have been proposed to investigate mainly the cerebral cortical formation (von der Malsburg, 1973; Amari & Takeuchi, 1978; Willshaw & von der Malsburg, 1979; Bienenstock, Cooper, & Munro, 1982; Obermayer, Ritter, & Schulten, 1990; Földiák, 1991).”

“input translations [10] it was used with (two layers of) linear units in which case it is only possible to learn invariances if there is little overlap between patterns [11] and the node will incorrectly respond if the input consists of partial patterns from different translational positions.”

“It has been proposed that different views of each object become associated with one another when they are experienced in succession during rotation (Földiák, 1991; Wiskott and Sejnowski, 2002; Wyss et al., 2006; Masquelier and Thorpe, 2007).”

“Experience with the statistical regularities in natural scenes has been proposed as a potential solution to this problem (Foldiak 1991; Ullman and Soloviev 1999; Wallis and Bu« lthoff 2001; Wallis and Rolls 1997; Wiskott and Sejnowski 2002).”

“• Fukushima [22] and Földiák [19] suggest that units within each level of the hierarchy form composite features from the conjunction of simpler features at lower levels to achieve shift-and scale-invariant pattern recognition.”

“By the principles of slow feature analysis ( 37 – 39 ), slow change in inputs that consist principally of sparse edge patterns would be expected to be optimal for a system learning to extract edges.”

“Several learning algorithms have been proposed that can achieve invariance learning (e.g., Foldiák, 1991; Stringer et al., 2006; Wiskott & Sejnowski, 2002), most of them based on the same general idea: objects in the world tend to remain present for several seconds, but changes in the position of…”

“While spatial statistics can explain well position or phase invariance in early vision 32,42,43 , temporal coherence 44 for learning the most slowly changing features has been commonly used for explaining more complex invariance properties in higher vision 36,45,46 and experimentally tested 47 .”

“This is a case of symmetry breaking by learning, which is prominently investigated in connectionist modelling (e.g., Földiák 1991).”

“…different views of the same object and ⁄ or experiencing their successive appearance may be necessary for recognition across larger changes in viewing angle (Foldiak, 1991; Wallis & Bülthoff, 2001; Wiskott & Sejnowski, 2002; Stringer & Rolls, 2006; Wyss et al., 2006; Masquelier & Thorpe, 2007).”

“Since temporal structure is known to help acquire invariance, we plan to incorporate trace learning into our learning algorithm (F¨oldiak, 1991; Hurri & Hyv¨arinen, 2003; Stringer, Rolls, & Tromans, 2007; Cadieu & Olshausen, 2009).”

“Therefore, for example, we will try to apply the trace rule [29] that is a modified Hebbian rule and changes synaptic weights according to both the current”

“The potential of Gibson's approach has recently begun to be realised as a series of connectionist models (Becker & Hinton, 1992; Becker, 1992; Zemel & Hinton, 1991; Foldiak, 1991; Schraudolph & Sejnowski, 1991; Mitchison, 1991; Phillips, Kay, & Smyth, 1995).”

“Another intriguing possibility is that such extended lingering of postsynaptic activity may reflect the existence of mechanisms allowing higher-order neurons to deploy the “trace” learning rule at the base of UTL [10] over longer integration times.”