Learning Invariance from Transformation Sequences

“There clearly exist, however, a variety of spatial and temporal contextual cues such as body pose and continuity of pose change which could be used [1, 10, 11, 22].”

“Unsupervised learning is the main mechanism: for a “complex” cell to pool over several simple cells, the key is a modified Hebbian rule based on temporal association (Foldiak-type rule, [30]): cells that fire in temporal contiguity are wired together.”

“…by a number of image modeling researchers (Chen & Gopinath, 2000; Shan, Zhang, & Cottrell, 2007; Lyu & Simoncelli, 2The linear relationship between the time derivative of phase and motion is a different effect from the linearization of the joint distribution dependencies of phase derivatives.”

“The “trace” rule of mechanism A1, for example, has been proposed long ago as a mechanism to set up invariant representations (Foldiak, 1991), and, while its asymmetry was experimentally characterized more precisely as STDP, it was proposed to underlie the learning of temporal sequences (Minai and…”

“Continuous transform (CT) learning is compared with another approach to invariance learning, trace learning, in this paper, and we summarise next the trace learning procedure developed and analysed previously (Földiák 1991; Rolls 1992; Rolls and Milward 2000; Rolls and Stringer 2001; Wallis and Rolls 1997).”

“invariances by tracking stable and slow-changing information in time (Földiák, 2008; Wiskott & Sejnowski, 2002).”

“that important stimulus features such as object category, the position of objects, or their orientation can be easily extracted by supervised training with high precision from the slow features of a high-dimensional visual input stream. It should be noted that the SFA algorithm is only one particular implementation of…”

“By presenting consistent sequences of transforming objects the cells in the network can learn to respond to the same object through all of its naturally transformed states, as described by Földiák (1991) and Rolls (1992).”

“This ‘‘ slowness ’’ or ‘‘ temporal stability ’’ principle is the basis of a whole class of learning algorithms [1–7].”

“This approach to unsupervised learning of invariant representations has been taken by a number of researchers since the early 90s (Földiák, 1991; Mitchison, 1991; Becker & Hinton, 1992; O'Reilly & Johnson, 1994; Stone & Bray, 1995; Wallis & Rolls, 1997; Peng et al., 1998; Kayser et al., 2001;…”

“One unsupervised learning principle in neuroscience is to minimize temporal variations created by motion in order to learn stable representations of object undergoing motion [27, 14, 24].”

“With such weights, this measure of slowness is similar to the objective of temporal smoothness used by Stone (1995) and somewhat related to the trace learning rules introduced by F¨oldi´ak (1991).”

“However, a simpler form of trace learning rule, involving only two firing rate terms, has been previously used as a biologically plausible learning rule for invariant object recognition (Földiák, 1991; Rolls & Deco, 2002; Wallis & Rolls, 1997).”

“The long persistence of the responses in STSa may also have another function (e.g., Foldiak, 1991 ).”

“It has been assumed that different views of each object become associated with one another during object rotation, either through active learning or through passive experiencing of the successive appearance of nearby views ( Foldiak 1991; Stryker 1991).”

“Use of such a rule has been proposed to be one way in which networks can learn invariant responses (FoÈ ldia k, 1991; Rolls, 1992; Wallis and Rolls, 1997; Rolls and Treves, 1998; Rolls and Milward, 2000; Rolls and Stringer, 2000).”

“This potentially simplifies readout from downstream neurons (Berkes & Wiskott, 2005; Földiák, 1991; C. Kayser, Körding, & König, 2003; Christoph Kayser, Einhäuser, Dümmer, König, & Körding, 2001; Körding, Kayser, Einhäuser, & König, 2004; Rolls, 2012; Wiskott & Sejnowski, 2002).”

“We conjectured that this could be done by unsupervised learning based on the temporal adjacency assumption [55, 56].”

“Besides redundancy reduction, a variety of other interesting criteria such as sparseness [7,8], temporal coherence [9], predictive information [10,11] , or bottom-up saliency [12] have been formulated.”

“As pointed out by Hinton and Sejnowski (1999), the generative framework can also be used to characterize some unsupervised STCNs, such as those involved in extracting perceptual invariances from spatiotemporal sequences of images (Földiák, 1991; Stone & Bray, 1995; Stone, 1996; Wallis, 1996; Kohonen, 1996; Kohonen, Ka…”

“2 Theory of Learning The idea that incorporating a trace of cell activity could aid the learning of natural transforms of objects was first discussed in detail by Foldi6k (FoldiAk 1991).”

“Hinton and Becker (1990) investigated a rule whereby units with nonoverlapping receptive fields try to maximize their mutual information.”

“…have long theorized that biological visual systems leverage the temporal smoothness of natural visual environments to recognize objects (e.g., DiCarlo, Zoccolan, & Rust, 2012; Feldman & Tremoulet, 2006; Foldiak, 1991; Gibson, 1979; Stone, 1996; Wallis & Rolls, 1997; Wiskott & Sejnowski, 2002).”

“, for V1-like complex units, learning which afferent units with the same orientation and neighboring locations should be wired together because, often, such a pattern changes smoothly in time (under translation) [31].”

“Biologically plausible models have been described for the creation of such units, that emerge by simple synaptic modification rules as a result of patterns drifting across the retina (Földiák, 1991; Parga & Rolls, 1998).”

“The slowness principle has been applied in many models of the visual system in the past [11, 12, 13, 5, 6, 2].”

“Foldiak (1991) showed that translation-invariance learning over a one-dimensional input array was possible for a simple winner-take-all network with a decaying trace of previous neuronal activity and an associative Hebb rule. Rolls (1992, 1994, 1995) and (Rolls & Treves, 1997) suggest that invariant representations co…”

“where kron denotes the Kronecker product (or tensor product) ( Eves, 1980 ) (see the appendix for definition),Ci = (Ii − I0i )IT 0i and Di = I0iI T 0i . Reshaping”

“(Földiak, 1991; Wallis & Rolls, 1997; O’Reilly & Johnson, 1994; Stone & Bray, 1995; Einhäuser, Hipp, Eggert, Körner, & König, 2005).”

“On a computational level, the idea that the brain uses temporal correlations to extract invariant information during sensory representation learning is the basis for slow feature analysis (33) and related approaches based on temporal stabilization (34, 35).”

“It is assumed that this process, which takes place gradually from V1 to TE, obeys the Hebbian learning rule, and includes a short memory trace of preceding neuronal activity (Foldiak 1991; Rolls 1992; Wallis and Rolls 1997).”

“"decorrelation" of Foldiak 1991 ) will automatically extract orthogonal dimensions (or principal components amongst sensory inputs).”

“Thanks also to all of the Topographica developers, without whom this work would not have been possible. and Sejnowski, 2002) or the trace rule (Földiák, 1991).”

“The algorithm is inspired by previous work [3] that exploits a simple trace rule for the learning of shift invariance.”

“A temporal stability objective has been used in some other unsupervised learning techniques (Hinton, 1989; Földiák, 1991; Mitchison, 1991; Schmidhuber, 1992a; Bergstra and Bengio, 2009).”

“A biologically plausible model for tracking and associating corresponding image regions was proposed by Foldiak (1991) and used in the VisNet model (Stringer & Rolls, 2002).”

“One conjecture is that temporal coherence [3, 13, 53] is the key 1This only gives a coarse approximation to the true neuronal connectivity and the actual information processing pathway.”

“It has been assumed that different views of each object become associated with one another during object rotation, either through active learning or through passive experiencing of the successive appearance of nearby view”

“…Swindale, 1996; von der Malsburg, 1973; Oja, 1989; Sanger, 1989; O’Reilly, 1998) or implicitly through a selection process which chooses the ‘win-ning’ node(s) (Rumelhart and Zipser, 1985; Kohonen, 1997; Grossberg, 1987; Ritter et al., 1992; Hertz et al., 1991; F ¨ oldi ´ ak, 1991; Wallis, 1996).”

“The ventral stream could take advantage of this natural tendency and learn to associate neuronal representations that occur closely in time to yield tolerant object selectivity (i.e., optimizing the “slowness” of visual representation) (Foldiak, 1991; Stryker, 1991; Wiskott and Sejnowski, 2002).”

“For simple visual features, like oriented lines, invariant detectors may even be built into the brain, much in the same way as complex cells pool information from several position-specific simple cells in early visual processing stages (Hubel & Wiesel 1962; Földiák 1991).”

“Such rules are able to build associations between different patterns of neural activities that tend to occur in temporal proximity (Földiák, 1991; Rolls & Stringer, 2001; Wallis and Rolls, 1997).”

“We then recorded responses to (1) sinusoidal gratings, (2) modified sparse-noise stimuli, and (3) static frames from natural and unnatural movie clips.”

“Foldiak (1991) has proposed computing an average activation of the postsynaptic neuron to assist with the same problem .”

“One proposed method for the learning of invariance in the visual system is to utilize the temporal continuity of objects in the visual environment (over short time periods) to help the learning of invariant representations (Földiák, 1991; Rolls, 1992; Wallis & Rolls, 1997; Rolls & Milward, 2000; Rolls & Stringer, 2001…”

“Accounting for cycle consistency [13, 30, 41] or temporal continuity [9, 43] in videos is another way to obtain a reliable proxy signal to learn correspondences without full supervision, or even to learn generic visual features [10, 40].”

“However, for clarity the left type of illustration will be used in the following. objective of extracting slowly changing features from the quickly varying sensory input (F¨oldi´ak, 1991), which then leads to a robust and invariant representation of the environment.”

“Under the temporal stability objective, neuronal RFs are adapted so that their output firing rates vary slowly in time [24–26].”

“…emerging from local plasticity rules implementing principal/independent component analysis [Oja, 1982, Hyvärinen and Oja, 1998], sparse coding [Olshausen and Field, 1997, Rozell et al., 2008] or slow-feature analysis [Földiák, 1991, Wiskott and Sejnowski, 2002, Sprekeler et al., 2007].”

“The first one are the so called “trace rule” models (Földiák 1991; Wallis and Rolls 1997; Rolls and Milward 2000; Stringer and Rolls 2000; Einhäuser et al. 2002; Spratling 2005; Masquelier et al. 2007), able to learn invariance (for e.g. to position and scale) from a series of frames in which…”

“A plausible and robust solution appears to be that the visual system associates views on the basis of their temporal proximity as well as spatial similarity (Pitts and McCulloch, 1947; Földiák, 1991; Miyashita, 1993; Wallis and Bülthoff, 1999; Wallis et al., 2009).”

“They crucially assumed that the content variable remains similar in consecutive frames, as inspired by the temporal coherence principle [Földiák, 1991].”

“With the SlowVAE model we provide a parsimonious solution to identifiable nonlinear ICA that is inspired by a long history of learning visual representations from temporal data [39, 40].”

“First, various interesting bottom-up approaches to hierarchical representation are based on the idea that higher-level structure changes more slowly than low-level structure (Földiak, 1991; Wallis & Baddeley, 1997; Becker, 1999; Laurenz & Sejnowski, 2002; Einhäuser, Kayser, König, & Körding, 2002; Körding, Kayser, Ein…”