Learning Invariance from Transformation Sequences

“The first description was given by Hinton [2], but implemented by Mitchison [3] and F6ldiak [4].”

“…should be noted that history-based rules can also be defined without changing the qualitative dynamics of the system: instead of considering the instantaneous value of the neurons’ activity, these rules consider its weighted mean overa time window (see [Földiák, 1991, Wallis and Baddeley, 1997]).”

“Natural stimuli are spatially and temporally correlated on a range of scales [46], and there is substantial evidence that temporal structure in extrinsic drive variables may be important for both neural coding and learning [47,48,49,50].”

“Both theoretical insights and experimental evidence have suggested how this wiring across views might be learned in development [18]–[21].”

“…of approaches, slow feature analysis ( Wiskott and Sejnowski, 2002 ) (SFA) and slow subspace analysis ( Kayser et al., 2001 ), stem from the idea of temporal coherence ( Földiák, 1991 ), which suggests that a key goal of sensory processing is to identify slowly varying features of natural inputs.”

“The activity ¯ v i of the post-synaptic cell is calculated by the temporal trace rule ¯ v t i = (1 − λ) ¯ v t−1 i + λv t i , 0 λ 1 (Földiák, 1991).”

“This kind of behavior has been the main idea behind many studies on spatiotemporal analysis of data, for example, spatial invariance learning (Földiák, 1991; Einhäuser, Kayser, König, & Körding, 2002; Hyvärinen, Hurri, & Väyrynen, 2003; Karklin & Lewicki, 2005), independent feature subspaces (Hyvärinen & Hoyer, 2000),…”

“SFA (Wiskott & Sejnowski, 2002) belongs to a family of algorithms for learning invariant representations from temporal input sequences, which maximize the “slowness” of their output signals (Földiák, 1991; Mitchison, 1991; Becker & Hinton, 1992; Stone & Bray, 1995).”

“A range of mathematical implementations of learning rules (Berkes & Wiskott, 2005; Földiák, 1991; Isik, Leibo, & Poggio, 2012; Körding, Kayser, Einhäuser, & König, 2004; Wiskott & Sejnowski, 2002) all include variants of this same conceptual idea: to achieve response stability of each neuron over…”

“The second candidate objective is called temporal stability, or slowness [26].”

“…(x t )] = 0 (Zero Mean) (2) E t [y 2 i (x t )] = 1 (Unit Variance) (3) E t [y i (x t )y j (x t )] = 0, ∀j = i (Decorrelation) (4) ∀j > i : s(y i ) ≤ s(y j ) (Order) (5) The principle of slowness, although not the above definition, has been used very early in the context of neural networks [2] [8].”

“are mutually dierent, we can identify Q up to a rotation within each block by computing the eigenvalue decomposition of (Y X (Y X )T)2 = Q(G GT)2QT . ( 17 )”

“…unsupervised learning algorithms that suggest that neural systems might develop encodings that reflect the characteristics of the input from the environment by maximizing simple criteria defined on the neural responses, for example, slowness (Foldiak, 1991) or sparseness (Olshausen, & Field, 1996).”

“The concept of temporal slowness ensures that neurons encode the slowly varying component of the signal [12].”

“+γy(t)) wi(t)→ r(t) xi(t) Földiák ẇi = ηθy(xi − wi) [16], [17] θ̇y = (y − θy) y → ||X|| any X”

“Other neuron-level models of invariance-learning (LeCun et al., 1989; Földiák, 1991; Rolls, 2012; Halvagal and Zenke, 2022) neither account for such feedback nor experimentally observed explicit encoding of mismatch between prediction and observation (Zmarz and Keller, 2016; Leinweber et al., 2017)…”

“Learning only occurs at the winning node at each iteration, and follows the general Hebb rule for synaptic modification [ 11 ].”

“…i.e., designing the representation itself to be immune to transformations, is a common approach in AI (Kazhdan, Funkhouser, & Rusinkiewicz, 2003; Foldiak, 1991), and aligns with findings from cognitive science that transformationinvariant properties exist in human mental representations…”

“…; Y 2 ) , then the pattern 1 2 In the example model of the visual system presented in the section above, the X neurons represent the last layer of the FH1 feature hierarchy whose neural connectivity was presumably produced through self-organized learning (e.g., Foldiak, 1991; Wallis et al., 1993).”

“A powerful and commonly used approach to feature extraction is to use multiple layers of hierarchically arranged feed-forward networks that implements competitive learning [3-6, 18-25].”

“A simple but naive way to achieve such invariance would be to list all the possible realizations of objects under various transformations, but this is not usually practical due to the vast amounts of possible realizations.”

“…visual experience of natural objects contains numerous variants that transform depending on the viewing conditions, it has been suggested that the capability to detect objects invariantly can develop gradually when observers repeatedly see objects with a wide range of variations (Földiák, 1991).”

“(Note that this differs from the global “winner-take-all” paradigm implemented by Földiák (1991). The cell activation is then passed through a nonlinear cell activation function, which also produces contrast enhancement of the firing rates (Rolls 2008b; Rolls and Deco 2002; Rolls and Milward 2000; Rolls and Stringer 2…”

“The trace rule is a proposed learning rule that associates instantaneous input with temporally filtered activation (F¨oldi´ak 1991).”

“Attempts thus far to incorporate other biologically-inspired concepts like invariances based on slowly-changing features (F¨oldi´ak, 1991; Wiskott and Sejnowski, 2002) have met with limited success.”

“To be specific, video content contains rich available and salient motion information between frames, which is essential for the human visual system to comprehend the structure of a moving object [10], [11].”

“Motion-based equivalence has been used to deal with the problem of position invariance [7] as well as more complex transformations [17],[19],[20].”

“• A modified Hebb-like learning rule incorporating a temporal trace of each cell’s previous activity, which, it is suggested (Földiák, 1991; Rolls, 1992, 2012; Wallis et al., 1993; Wallis and Rolls, 1997), will enable the neurons to learn transform invariances.”

“Our findings highlight the need to develop processes that can support more robust translation tolerance (e.g., Földiák, 1991) that have been omitted in most current theories and models of object and word recognition.”

“), the underlying assumption of these approaches is that relevant properties typically vary on a slower time-scale than irrelevant ones (Földiak 1991; Körding and König 2001; Wiskott and Sejnowski 2002).”

“Invariances can, to a limited degree, be learned from real-world data based on the assumption that temporally continuous sequences leave the object identity unchanged (Földiák, 1991; Hinton, 1987; Bartlett and Sejnowski, 1998; Wiskott and Sejnowski, 2002).”

“…that neurons remove statistical redundancies but in this case by signalling unpredictable features within sensory input [70–73]; temporal coherence [74], slow subspace analysis [75] and slow feature analysis [76, 77], which hypothesize that V1 encodes slowly-varying features within the stimulus;…”

“Földiák (1991) introduced a local synaptic learning rule which could lead to translation-invariant responses in a schematic model of complex cells in visual cortex.”

“…& Hyvärinen, 2003) and complex cells (Berkes & Wiskott, 2005; but also see Lies, Häfner, & Bethge, 2014) of primary visual cortex as well as different visual invariances (Földiák, 1991; Mitchison, 1991; O’Reilly & Johnson, 1994; Wallis & Rolls, 1997; Einhäuser, Hipp, Eggert, Körner, & König, 2005).”

“Our approach to learning abstract representations from multimodal data can also be related to approaches that learn view-invariant visual object representations through temporal coherence [15], [16].”

“The actual learning rule used here was originally proposed by Sutton and Barto [24], although its relevance to object recognition learning and specifically in setting up representations in the primate temporal lobe, is more recent [17], [18].”

“The most prominent are sparse coding [1] and temporal coherence [2,3].”

“An alternative is to learn invariance, for example by augmenting the training set with perturbations of the training set [1, 3, 27], through temporal cues [8, 26, 20] or incorporating linear transformation operators into feature learning algorithms [33].”

“These parameters develop according to synaptic and homeostatic learning rules inspired by sparse coding theory and previously established rules such as Oja’s Hebbian plasticity rule [42] and Foldiak’s rule [43].”

“…learning with the additional constraint of an activity trace, often modeled by using a (artificial) history of previous activations in exchange of the activation in Hebb-type learning rules (F¨oldi´ak, 1991; Wallis & Rolls, 1997; Einh¨auser, Kayser, K¨onig, & K¨ording, 2002; Spratling, 2005).”

“The training episode captures the idea that temporally neighboring inputs are similar, and that the resulting representation should vary slowly, and thus is related to ideas of slow feature analysis [8].”

“Recently, more work has begun to explore the possibility of ‘learned’ equivariance guided by the data itself [4, 14, 20].”

“This is in contrast to other algorithms for learning invariance which require stimuli to be presented in isolation e.g., [30], [59], [91], [97], and [101].”

“One simple heuristic for measuring the usefulness of data might be temporal smoothness or stability (Földiak 1991; Kayser, Einhäuser et al. 2001).”

“Dynamic stimuli and temporal learning rules: There is a long history of using the temporal structure of visual stimuli to extract transformation-invariant representations [Földiák, 1991, Wallis and Rolls, 1997, Masquelier et al., 2007].”

“Furthermore, the temporality of the scene is explicitly maintained all the way into the internal video representation: SFA codes are threaded along frames, such that local regions are represented with temporal outputs of size .”

“During training with visual objects, while the connectivity pattern is fixed, the strengths of the feedforward and feedback synaptic connections between successive neuronal layers are modified by a trace learning rule (Foldiak, 1991; Wallis & Rolls, 1997), which incorporates a memory trace of recent neuronal activity:”

“Earlier studies using the temporal coherence objective to learn invariant responses delivered a clear proof of concept using artificial (Földiak 1991) as well as artificially transformed natural stimuli (Stone 1996; Wallis & Rolls 1997).”

“Accordingly, our proposed solution is temporal trace learning (Foldiak, 1991; Wallis and Rolls, 1997; Rolls and Milward, 2000).”

“Two examples for general principles are ‘slowness’ [19] for view-invariant object recognition and ‘universal prediction tasks’ [3] for the autonomous generation of abstract concepts.”

“This could potentially be achieved using an unsupervised learning rule that forms associations across time (Földiák, 1991; O’Reilly & McClelland, 1992; Spratling, 2005; Templeman & Loew, 1989; Wallis, 1996).”