Local minimization of prediction errors drives learning of invariant object representations in a generative network model of visual perception

“To construct useful representations, biological systems are limited to mostly unsupervised learning (from unlabeled data) and local learning rules, whereas machine vision algorithms based on neural networks typically rely on large amounts of labeled training data and use mechanisms like weight-sharing (LeCun et al., 1…”

“Strengthening theories of generative perception (Gregory, 1980; Mumford, 1992; Rao and Ballard, 1999; Friston, 2010; Pennartz et al., 2019), evidence has accumulated to suggest that the mammalian perceptual OPEN ACCESS”

“The more detailed and sample-specific information may provide useful input to the action-oriented dorsal processing stream (Goodale and Milner, 1992), whereas the hierarchy of the ventral visual cortex extracts object identity and relevant concepts (Mishkin et al.”

“The Hebbian learning rule thus groups together consecutive inputs in a manner reminiscent of contrastive, self-supervised methods (34,35,58) 275”

“Strictly symmetric weight matrices as frequently used in predictive coding models (5,18) lead to a weight transport problem during learning.”

“Evidence for invariant neural representations comes from the ventral temporal lobe (21), the hippocampus in humans (22), inferotemporal cortex of rhesus (23,24) and macaque monkeys (25) as well as rats’ laterolateral extrastriate area (LL) (26,27).”

“Here, combination of receptive field-like local filters with a pooling mechanism (Riesenhuber and Poggio, 1999) may be helpful to become tolerant to the varying configurations of individual features comprising the objects from the same class.”

“This repetition of individual sequences drastically improved network performance and could be achieved by the brain through a replay or reactivation mechanism (observed in visual cortex by Ji and Wilson, 2007 and Xu et al., 2012, see also Wilson and McNaughton, 1994; Lansink et al., 2009).”

“…in superficial, and representations in deep cortical layers 10.3389/fncom.2023.1207361 Frontiers in Computational Neuroscience 13 frontiersin.org (Bastos et al., 2012; Keller and Mrsic-Flogel, 2018; Pennartz et al., 2019; Jordan and Keller, 2020), area-and layer-wise recordings of characteristic…”

“Evidence for invariant neural representations comes from the ventral temporal lobe (Haxby et al., 2001), the hippocampus in humans (Quiroga et al., 2005), inferotemporal cortex of rhesus (Desimone et al., 1984; Logothetis et al., 1995) and macaque monkeys (Freiwald and Tsao, 2010) as well as rats’…”

“This repetition of individual sequences drastically improved network performance and could be achieved by the brain through a replay or reactivation mechanism (observed in visual cortex by Ji and Wilson, 2007 and Xu et al., 2012, see also Wilson and McNaughton, 1994; Lansink et al., 2009).”

“According to predictive coding theory, these predictions are mediated by corticocortical top-down connections (Pennartz et al., 2019) and then corrected based on the received bottom-up input (Rao and Ballard, 1999) in line with hierarchical Bayesian perception (Lee and Mumford, 2003).”

“Evidence for invariant neural representations comes from the ventral temporal lobe (21), the hippocampus in humans (22), inferotemporal cortex of rhesus (23,24) and macaque monkeys (25) as well as rats’ laterolateral extrastriate area (LL) (26,27).”

“Furthermore, it is intimately related to the abovementioned slowness principle, which states that the most meaningful features often change on a slow timescale (Wiskott and Sejnowski, 2002), because extracted causes tend to be good predictors for future input (Creutzig and Sprekeler, 2008).”

“Strengthening theories of generative perception (Gregory, 1980; Mumford, 1992; Rao and Ballard, 1999; Friston, 2010; Pennartz et al., 2019), evidence has accumulated to suggest that the mammalian perceptual Brucklacher et al. 10.3389/fncom.2023.1207361 Frontiers in Computational Neuroscience 02…”

“of neural responses by expectations based on the surrounding context (14) show that predictions are not only made forward in time, but also across space (in the present).”

“While there is ample evidence for a hierarchy of timescales in the visual processing streams of humans (Hasson et al., 2008), primates (Murray et al., 2014) and rodents (Piasini et al., 2021), with larger temporal stability in higher areas, the compatibility with deep predictive coding is debated…”

“, 2005), inferotemporal cortex of rhesus (Desimone et al., 1984; Logothetis et al., 1995) and macaque monkeys (Freiwald and Tsao, 2010) as well as rats’ laterolateral extrastriate area (LL) (Tafazoli et al.”

“While there is ample evidence for a hierarchy of timescales in the visual processing streams of humans (Hasson et al., 2008), primates (Murray et al.”

“In combination with work on encoding of errors in superficial, and representations in deep cortical layers (3,63,76,77), area- and layer-wise recordings of characteristic timescales could lead to a better understanding of cortical microcircuits underlying predictive coding.”

“Most predictive coding models so far, however, either operate on static inputs (5,18,41) or use non-local learning rules (42) such as backpropagation (43,44) and biologically implausible LSTM units (16,45).”

“from the work of (51) that showed how Hebbian plasticity regulates inhibitory input to reduce firing and achieve a balanced global state.”

“Layer-wise investigations also show distinct patterns of feedforward and feedback connectivity (Markov et al., 2014) and information processing (Oude Lohuis et al., 2022).”

“Other neuron-level models of invariance-learning (LeCun et al., 1989; Földiák, 1991; Rolls, 2012; Halvagal and Zenke, 2022) neither account for such feedback nor experimentally observed explicit encoding of mismatch between prediction and observation (Zmarz and Keller, 2016; Leinweber et al., 2017)…”

“This repetition of individual sequences drastically improved network performance and could be achieved by the brain through a replay or reactivation mechanism (observed in visual cortex by Ji and Wilson, 2007 and Xu et al., 2012, see also Wilson and McNaughton, 1994; Lansink et al., 2009).”

“Across time, repetition suppression that requires explicit expectations (Summerfield et al., 2008; Todorovic et al., 2011), encoding of deviation from temporal expectations in macaque’s inferotemporal and prefrontal cortex (Schwiedrzik and Freiwald, 2017; Bellet et al., 2021) and encoding of…”

“This comparison of distinct subpopulations may provide an additional angle to measuring neural correlates of prediction errors [for a review see (Walsh et al., 2020)], as representation neuron responses have been barely considered in experimental work so far.”

“…as an implicit objective function, upon which the inference steps conduct an approximate gradient descent taking into account only the sign and not value of the derivative of the activation function, unlike (Whittington and Bogacz, 2017), and upon which learning conducts a precise gradient descent.”

“Indeed there is evidence for the importance of temporal stimulus continuity for learning of transformation-tolerance in early visual areas of rats (Matteucci and Zoccolan, 2020) and area IT of monkeys (Li and DiCarlo, 2008).”

“, 2018) and information carrying (Smith and Muckli, 2010) feedback to early visual areas (cf.”

“Here, the higher-level representation from the previous timestep provides a target for the consecutive inputs reminiscent of implementations of supervised learning with local learning rules (Lee et al., 2015; Whittington and Bogacz, 2017; Haider et al., 2021).”

“In combination with work on encoding of errors in superficial, and representations in deep cortical layers (3,63,76,77), area- and layer-wise recordings of characteristic timescales could lead to a better understanding of cortical microcircuits underlying predictive coding.”

“This repetition of individual sequences drastically improved network performance and could be achieved by the brain through a replay or reactivation mechanism (observed in visual cortex by (53,54), see also (55,56)).”

“9 Some models such as (44) suggest computation of errors in dendrites, but based on the evidence for neural encoding of errors (35), we assign dedicated neurons to encode them.”

“Other neuron-level models of invariance-learning (LeCun et al., 1989; Földiák, 1991; Rolls, 2012; Halvagal and Zenke, 2022) neither account for such feedback nor experimentally observed explicit encoding of mismatch between prediction and observation (Zmarz and Keller, 2016; Leinweber et al., 2017)…”

“9 Some models such as (47) suggest computation of errors in dendrites, but based on the 150”

“time, repetition suppression that requires explicit expectations (6,7), encoding of deviation from temporal expectations in macaque’s inferotemporal and prefrontal cortex (8,9) and encoding of expected movement outcomes in mouse V1 (10) show that the brain constantly tries to predict future inputs.”

“…hierarchies purely as a result of dynamics in spiking neurons (van Meegen and van Albada, 2021) or large-scale models (Chaudhuri et al., 2015; Mejias and Wang, 2022), our model provides a complementary account, postulating development of the temporal hierarchy as a consequence of a…”

“Predictive coding models have successfully explained properties of the visual system such as end-stopping in V1 neurons and learning of wavelet-like receptive fields (Rao and Ballard, 1999) and V1 activity in illusory contours (Lotter et al., 2020; Pang et al., 2021).”

“Conversely, auto-associative Hopfield-type models that learn dynamic pattern completion from local learning rules (Herz et al., 1989; Brea et al., 2013) do not learn hierarchical invariant representations like the proposed model does.”

“…the model acquired generative capacity that enables reconstruction of partially occluded stimuli, in line with retinotopic and content-carrying feedback connections to V1 (Smith and Muckli, 2010; Marques et al., 2018), see also (Pennartz et al., 2019) for a review of predictive feedback mechanisms.”

“Conversely, auto-associative Hopfield-type models that learn dynamic pattern completion from local learning rules (Herz et al., 1989; Brea et al., 2013) do not learn hierarchical invariant representations like the proposed model does.”

“inferotemporal cortex of rhesus (23,24) and macaque monkeys (25) as well as rats’ laterolateral extrastriate area (LL) (26,27).”

“…these models process information in a strictly feedforward manner or limit the role of feedback connections to a modulatory function, in contrast to evidence on retinotopic, content-carrying feedback connections in the visual cortex (Zmarz and Keller, 2016; Marques et al., 2018; Pak et al., 2020).”

“The model also outperformed linear Slow Feature Analysis (SFA) (Wiskott and Sejnowski, 2002) of the raw inputs (for details see Supplementary material 1.12).”

“At the same time, the model acquired generative capacity that enables reconstruction of partially occluded stimuli, in line with retinotopic and content-carrying feedback connections to V1 ((13,38), see also (3) for a review of predictive feedback mechanisms).”

“self-supervised methods (35,36,59) that explicitly penalize dissimilarity in the loss function.”

“Most predictive coding models so far, however, either operate on static inputs (5,18,41) or use non-local learning rules (42) such as backpropagation (43,44) and biologically implausible LSTM units (16,45).”

“inferotemporal cortex of rhesus (23,24) and macaque monkeys (25) as well as rats’ laterolateral extrastriate area (LL) (26,27).”

“Furthermore, it is intimately related to the abovementioned slowness principle, which states that the most meaningful features often change on a slow timescale (Wiskott and Sejnowski, 2002), because extracted causes tend to be good predictors for future input (Creutzig and Sprekeler, 2008).”

“While there is ample evidence for a hierarchy of timescales in the visual processing streams of humans (70), primates (71) and rodents (62), with larger temporal stability in higher areas, the compatibility with deep”

“Other self-supervised models avoid representational collapse through contrasting examples (36).”

“Indeed there is evidence for the importance of temporal stimulus continuity for learning of transformation-tolerance in early visual areas of rats (Matteucci and Zoccolan, 2020) and area IT of monkeys (Li and DiCarlo, 2008).”

“A biologically plausible approach to learn view-invariance from transformation sequences is so-called trace learning (Földiák, 1991; Elliffe et al., 2000; Rolls, 2012) which is linked to Slow Feature Analysis (SFA) (Sprekeler et al., 2007).”

“Unlike (Dora et al., 2021), the present implementation also does not require weight regularization that depends on information not readily available at the synapses.”

“…the model generalizes predictive coding to moving stimuli in a biologically more plausible way than other approaches (Lotter et al., 2016, 2020; Ali et al., 2021) that rely on error backpropagation, which is non-local (Rumelhart et al., 1985) or the equivalently non-local backpropagation…”

“been shown that under constraints which would be out of the scope of this paper to discuss, predictive coding can do exact backpropagation when clamping the highest layer activities in a supervised manner (53,67).”

“of dynamics in spiking neurons (61) or large-scale models (62,63), our model provides a complementary account, postulating development of the temporal hierarchy as a consequence of a functional computation: learning invariance by local error minimization.”

“V1 activity evoked by illusory contours (Bartels, 2014; Kok and de Lange, 2014), encoding of information from occluded scene areas in early visual areas of humans (Smith and Muckli, 2010) and modulation of neural responses by expectations based on the surrounding context (Knierim and van Essen,…”

“However, it has been shown that the lack of inductive bias in fully connected models can be compensated for by training on larger amounts of data (Bachmann et al., 2023).”

“Most predictive coding models so far, however, either operate on static inputs (5,18,42) or use non-local learning rules (43) such as backpropagation (44,45) and biologically implausible 115”

“However, it has been shown that, in combination with weight decay, symmetric weights can be obtained by learning rule comparable to ours without explicitly enforcing symmetry (50), since the locally available pre- and postsynaptic activity that determine the weight change are identical (symmetric) for 170”

“Complementary computational approaches can be utilized to foster further comparison and integration, 116,188,196,197 and research is being extended from human to animal consciousness to benefit from in vivo two-photon imaging, large-scale ensemble recordings, complexity measures, and optogenetics to…”

“Importantly, PC employs local information to update synapses and can be implemented under biologically plausible constraints and learning rules [65,66], over-coming the disadvantages of models based on artificial learning rules.”

“…and more varied receptive field structures, another open avenue is to expand the functionality of the model to implement more sophisticated types of task, including object invariance properties to take into account visual rotations or transformation common in real-world applications [7].”