Learning Invariance from Transformation Sequences

“An approach that exploits this temporal contiguity is given by the trace learning rule (Földiák, 1991).”

“Slowness has been shown to be a plausible criterion for learning the invariance exhibited by complex cells in the primary visual cortex (Földiák, 1991).”

“When a sufficient number of views is learned and linked via temporal or spatiotemporal proximity and the tunings of the view-tuned units overlap, the entire view space of the object is represented (Földiák, 1991; Tian & GrillSpector, 2015; Wallis & Bülthoff, 2001).”

“…without any explicit teaching signal, the new CBM methodologies130 utilise two previously established biologically plausible synaptic learning mechanisms known as131 continuous transformation (CT) learning (Stringer et al., 2006) and trace learning (Foldiak, 1991;132 Wallis and Rolls, 1997).”

“Ideas borrowed from [26, 27] and [11, 9, 10] are used to modify weights between neurons in the same array.”

“Invariances can, to a limited degree, be learnt from real-world data based on the assumption that temporally continuous sequences leave the object identity unchanged [2,6,1,13].”

“In the context of recent progress in modelling temporal processes (Foldiak 1991; Becker 1996; Stone 1996a; additional demonstration that temporal smoothness can act as a powerful constraint on the nature of learning in artificial neural networks, and(cid:246)we conjecture(cid:246)in ‘real’ neural…”

“A trace rule [11] is used to maintain a recent movement history.”

“Peter Földiák further extended this work from simultaneous events to sequences through his trace conditioning model, demonstrating how neurons can build invariant representations based on temporal contiguity (Földiák 1991).”

“This is a common machine learning problem [11] [12] [13] [14].”

“A number of theoretical studies were reported to explain the properties of complex cells (Földiák, 1991; Hyvärinen and Hoyer, 2001; Berkes and Wiskott, 2005; Karklin and Lewicki, 2005, 2009; Shan et al., 2007).”

“An alternative to CT learning is trace learning, which has been applied to the problem of visual invariance before by Foldiak (1991) and Wallis et al. (1993).”

“An alternative formulation often attributed to Klopf [56] (as reported by [57]) is a slowness or stability criterion, where the aim is to minimise variation over time of the complex-cells’ responses [57–59].”

“In 1991, Földiák [2] analyzed the SFs based on online learning theory.”

“The activity v̄i of the postsynaptic cell is calculated by the temporal trace rule v̄ i = (1 − λ)v̄t−1 i + λv i [7], 0 < λ < 1.”

“The first description of temporal coherence principle was given by Hinton [13] as a learning rule in back-propagation neural networks, and it was implemented by Mitchison [14] and Földiák [15].”

“…A modi fi ed associative (Hebb-like) learning rule incorporating a temporal trace of each neuron ’ s previous activity, which, it has been shown (Földiák, 1991; Rolls, 1992, 2012, 2016; Rolls & Milward, 2000; Rolls & Stringer, 2001; Wallis & Rolls, 1997; Wallis et al., 1993), enables the…”

“This observation has led to the concept of slowness learning [10], [11], [12].”

“A different coding principle based on the trace rule originally proposed by Földiak [[3]], namely temporal coherence or stability, has also been shown to lead to the emergence of simple [[4]] and complex [[5]] type receptive fields when applied to natural image sequences.”

“changing features is essential for the development of vision (e.g., DiCarlo, Zoccolan, & Rust, 2012; Földiák, 1991; Rolls, 2012; Stone, 1996; Wiskott & Sejnowski, 2002).”

“The form of prior which penalizes temporal variability of the signal is long known [6, 27, 12, 25].”

“The slowness principle was probably first formulated by Hinton (1989), and online learning rules were developed shortly after by Földiák (1991) and Mitchison (1991).”

“This method of learning view-invariance is really just an extension of the ‘trace rule’ (Földiák, 1991; Wallis et al., 1993) but one that takes advantage of computations and behavioral knowledge available elsewhere in the brain.”

“Invariance to symmetries in pattern recognition is an old and challenging problem [1], [2], [3], [4], [5], [6], [7], [8], [9], [10], [11], [12].”

“Besides the temporal trace learning rule (Földiák, 1991), Slow Feature Analysis (SFA) has been proposed as one of the few other unsupervised frameworks to solve the invariance problem in our brains (Berkes & Wiskott, 2005; Wiskott & Sejnowski, 2002).”

“Finally, the first term in the synaptic weights update resembles the first term in the following local “trace rule” proposed by Földiák [93] for learning temporally invariant features: Wtrace ←Wtrace + η ( (zt + zt−1)x ⊤ t − diag(zt + zt−1)Wtrace ) .”

“Other theories have been proposed to account for the unsupervised discovery of unknown invariances, based on the temporal association of sequences of transformation-related patterns over short memory timescales (Foldiak, 1991; Wallis et al., 1993; Bartlett & Sejnowski, 1995).”

“Földiák (1991) demonstrate a similar result in simulation: with a local learning rule (minimizing the number of weights being updated), a predictive neural network learns invariances in temporal sequences and becomes able to do simple generalizations through groups of cells reacting to similar…”

“For these simulations we used a trace learning rule [25,26] to adjust the strengths of the feed-forward synaptic connections between the layers during training.”

“It is also possible that combining CT learning with a trace learning rule (Foldiak, 1991) could improve the ability of the network to form separate invariant representations of different objects.”

“We have hypothesised that head-centered visual neurons might develop in the primate parietal cortex through visually-guided learning by combining trace learning (Foldiak 1991) with the natural statistics of eye and head movement (Freedman and Sparks 1997; Einhäuser et al. 2007).”

“All rights reserved. doi:10.1016/j.tcs.2008.06.054 The second principle, the principle of slowness (introduced by Földiak [4], promoted and developed by Wiskott and Sejnowski [21]), states that sensory signals vary more quickly than relevant state properties.”

“C Hebbian kernel rule implementing PCA (Oja 1982) when τST DP → 0ms and equivalent to the trace learning (Földiák 1991) in general.”

“Alternatively, a trace learning rule (Foldiak, 1991; Wallis & Rolls, 1997) may be implemented, which incorporates a memory trace of recent neuronal activity: wij kr̄i 1rj (A7) where r̄ i is the trace value of the firing rate of postsynaptic neuron i at timestep .”

“Many existing unsupervised invariant feature extraction methods are based on temporal invariance [5]-[7]: it is a reasonable assumption that the identity of objects-such as the identity of a person-changes slower than sensory input-such as image on retina and sound to cochlea.”

“As a consequence of this construct, each Complex cell has an associated single Simple cell for which it is predicting; specifically, Complex cell #1 predicts the activity of Simple cell #1, Complex cell #2 predicts the activity of Simple cell #2, etc.”

“…(and potentially on the momentary state of the synapse itself) have been useful to describe the development of receptive fields (RFs) in models of V1 [15, 16, 17, 18, 19, 20, 21, 22, 23], but are not sufficient to explain the increase of abstraction found across the ventral visual stream [4].”

“A more complex issue is how our results relate to models that use temporal association between distinct object views to facilitate generalization across views via binding over time (Foldiak, 1991; Wallis, 1998).”

“For continuously varying sequences, Foldiak’s trace rule [10] is a natural choice for identifying such groups.”

“A trace is a kind of temporal average (or moving average) [ 8 ].”

“The effect of such a learning rule is to encourage individual postsynaptic neurons to learn to respond to subsets of input patterns that tend to occur close together in time [22,23].”

“In Section ‘‘Discussions’’, robust coding is compared with other efficient coding schemes, such as sparse coding (Olshausen and Field 1996, 1997; Simoncelli and Olshausen 2001) and temporal coherence (Földiák, 1991; Becker 1993; Stone 1996; Wiskott and Sejnowski 2002; Hurri and Hyvärinen 2003).”

“Many variants of the Hebbian learning explain the nature of neural representation and its relation to the coding mechanisms observed in biological neural tissue [15,26].”

“The idea that temporal correlation of visual events can promote invariance learning has been explored by a number of earlier neural models (Foldiak 1991, Wallis and Rolls 1997, Wiskott 2002), even though learning from video is still at its infancy in computer vision.”

“Temporal association methods like ours have previously been shown to work in simple uncluttered images [7, 9, 12, 13].”

“We previously showed that approximations to the hardwired invariance in the HMAX architecture can be learned from natural videos in an unsupervised manner by employing a temporal coherence principle [3, 8, 16].”

“Inspired by slow feature analysis [16, 25,27,55], we encourage adjacent views in the triplet to form a smooth trajectory in the feature space, implemented with a simple local linearity assumption.”

“Another relevant issue is the slowness regularizers (Hinton [7], Földiák [4], Luciw and Schmidhuber [13], Jonschkowski and Brock [10], Merity et al. [15]) which add a penalty for large changes in state through time.”

“This represents one example of a more general principle of exploiting the temporal structure of visual input to learn invariant representations [3]–[6].”

“Based on the argument by Livingstone and Hubel [10] that cells in the visual pathway are organized in a hierarchical way with increasing receptive field size and higher visual abstraction from lower to higher layers, Ungerleider and Mishkin [12] established a theory of two parallel visual pathways.”

“Several algorithms have been proposed for learning transformation sequences (e.g., Perrett et al., 1984; Foldiak, 1991; Hietanen et al., 1992; Wallis et al., 1993; Einhäuser et al., 2002; Wiskott and Sejnowski, 2002; Spratling, 2005; Stringer et al., 2006; Masquelier et al., 2007).”

“…models of this process have been implemented as well, lending credence to the idea that learning to recognize a novel object under different viewing conditions may be facilitated by associative learning over time (Foldiak, 1991; Stone, 1999; Stone & Harper, 1999; Ullman & Bart, 2004; Wallis, 1998).”

“…tuned to the presence of a visual target with respect to just one pose of the hand, while other neurons could develop pose invariant responses through an invariance learning mechanism such as trace learning (Földiák, 1991; Rolls, 1992) or continuous transformation learning (Stringer et al., 2006).”

“SFA is based on the slowness principle (Földiák, 1991; Mitchison, 1991; Wallis & Rolls, 1997), which states that the underlying causes of changing signals vary more slowly than the primary sensory stimulus.”

“Thecontinuing efforts to address the invariant object recognition issue also include some more recent approaches such as Foldiak (1991), Olshausen et al (1993), Wallis and Rolls (1997), and see Riesenhuber and Poggio (2000) for a review.”

“Alter-natively, a trace learning rule (Foldiak, 1991; Wallis & Rolls, 1997) may be implemented, which incorporates a memory trace of recent neuronal activity: where (cid:2) r i is the trace value of the firing rate of postsynaptic neuron i at timestep (cid”