The Dual Form of Neural Networks Revisited: Connecting Test Time Predictions to Training Patterns via Spotlights of Attention

“Also, while there is no technical gain in expressing the dual form in terms of key-value/attention concepts (Vaswani et al., 2017; Miller et al., 2016; Sukhbaatar et al., 2015; Graves et al., 2014), such a formulation has become very popular since the advent of Transformers (Vaswani et al., 2017), and we believe many…”

“We train language models based on long short-term memory recurrent NNs (Hochreiter & Schmidhuber, 1997) on a small public domain book and the WikiText-2 dataset (Merity et al., 2017).”

“A desirable NLP experiment would be to train a big Transformer language model (LM) on a vast amount of data, and analyse the attention of linear layers over the training datapoints while sampling from the model.”

“For example, how can big language models such as GPT-3 (Brown et al., 2020) generate an answer to a question never seen during training, translate languages without being trained to do so, or solve previously unseen math problems?”

“…presented in the case where the co-domain of the linear layer is one-dimensional (i.e., the weight matrix reduces to a weight vector), especially in the literature on support vector and kernel machines (Boser et al., 1992; Burges, 1998) and in textbooks (Sch¨olkopf & Smola, 2002; Bishop, 2006).”

“…exception of certain special structures such as filters in convolutional NNs trained for image processing (Zeiler & Fergus, 2014), attention weights (Bahdanau et al., 2015) or the sequential Jacobian (Graves, 2008) in sequence processing, or, to a limited extent, the distribution of individual…”

“…patterns, with the occasional exception of certain special structures such as filters in convolutional NNs trained for image processing (Zeiler & Fergus, 2014), attention weights (Bahdanau et al., 2015) or the sequential Jacobian (Graves, 2008) in sequence processing, or, to a limited…”

“Most importantly, it is well known that the perceptron (Rosenblatt, 1958) has primal and dual forms as pointed out by Aizerman et al. (1964).”

“…presented in the case where the co-domain of the linear layer is one-dimensional (i.e., the weight matrix reduces to a weight vector), especially in the literature on support vector and kernel machines (Boser et al., 1992; Burges, 1998) and in textbooks (Sch¨olkopf & Smola, 2002; Bishop, 2006).”

“Obviously, referring to the equations above as attention (or unnormalised attention) emphasizes the relation to the regular softmax normalised dot product attention (Luong et al., 2015; Bahdanau et al., 2015) which is V softmax( K (cid:124) q ) using the notations above.”

“For image classification, we analyse feedforward NNs with two hidden layers using the MNIST (LeCun et al., 1998) and Fashion-MNIST (Xiao et al., 2017) datasets.”

“Example test query and top-3 training passages (with their Wikipedia page title) from WikiText-2.”

“Naive visualisation of their activations or weight matrices rarely shows human-interpretable patterns, with the occasional exception of certain special structures such as filters in convolutional NNs trained for image processing (Zeiler & Fergus, 2014), attention weights (Bahdanau et al., 2015) or the sequential Jacobi…”

“, 2020) generate an answer to a question never seen during training, translate languages without being trained to do so, or solve previously unseen math problems? How can DALL-E (Ramesh et al., 2021) generate various pictures of a radish walking a dog or a banana performing stand-up comedy without training examples co…”

“0 keys for the attention computation, the model would be able to reproduce the good performance on MNIST obtained after the first part of training, and thus avoid degradation or more generally catastrophic forgetting (French, 1999).”

“…gain in expressing the dual form in terms of key-value/attention concepts (Vaswani et al., 2017; Miller et al., 2016; Sukhbaatar et al., 2015; Graves et al., 2014), such a formulation has become very popular since the advent of Transformers (Vaswani et al., 2017), and we believe many modern…”

“Also, while there is no technical gain in expressing the dual form in terms of key-value/attention concepts (Vaswani et al., 2017; Miller et al., 2016; Sukhbaatar et al., 2015; Graves et al., 2014), such a formulation has become very popular since the advent of Transformers (Vaswani et al., 2017),…”

“Katharopoulos et al. (2020) convert the self-attention in Transformers—whose complexity is quadratic in sequence length—to a linear-complexity linear layer with fast weights (Schmidhuber, 1991).”

“A number of recent works connect attention and kernels (Tsai et al., 2019; Katharopoulos et al., 2020; Choromanski et al., 2021; Peng et al., 2021).”

“…presented in the case where the co-domain of the linear layer is one-dimensional (i.e., the weight matrix reduces to a weight vector), especially in the literature on support vector and kernel machines (Boser et al., 1992; Burges, 1998) and in textbooks (Sch¨olkopf & Smola, 2002; Bishop, 2006).”

“3, the conversion from the primal to the dual form of a linear layer we explore here (Proposition 1) is used in prior works connecting Transformers with linearised attention and fast weight programmers (Ba et al., 2016; Katharopoulos et al., 2020; Schlag et al., 2021).”

“A number of recent works connect attention and kernels (Tsai et al., 2019; Katharopoulos et al., 2020; Choromanski et al., 2021; Peng et al., 2021).”

“13 remains invariant if we shuffle the order of columns in the key-value storage, as it is carried out without any explicit positional encoding (similarly to auto-regressive attention (Irie et al., 2019; Tsai et al., 2019) in some Transformer language models).”

“More recent work of Domingos (2020) presents a more general statement: an entire multi-layer perceptron can be approximated by a kernel machine.”

“3, the conversion from the primal to the dual form of a linear layer we explore here (Proposition 1) is used in prior works connecting Transformers with linearised attention and fast weight programmers (Ba et al., 2016; Katharopoulos et al., 2020; Schlag et al., 2021).”

“We also note that Schmidhuber (1993) referred to the unnormalised attention used in the context of fast weight controllers as “internal spotlights of attention.””

“A direct estimation method (Direct) adopts the conditional probability of candidate answers that can be represented by tokens in the vocabulary of language models (Brown et al., 2020).”

“Whisper utilizes the encoder of a Transformer model to derive features from audio input, which are then fed into the decoder through multi-headed cross-attention, enabling auto-regressive text token prediction (Wang et al., 2023; Irie et al., 2022).”

“Mechanistically, ICL has been linked to neural architectures, such as induction heads [24, 59] and functional modules that emerge naturally during training [7, 43, 81, 84].”

“Motivated by the inherent duality between In-Context Learning (ICL) and gradient descent (Dai et al., 2022; Aizerman et al., 1964; Yang et al., 2023; Irie et al., 2022), we “fine-tune” the instruction implicitly through one-shot learning, replacing the need for actually fine-tuning the model.”

“…delving into the inner mechanism (Olah, 2022; Olsson et al., 2022) of ICL through different theoretical conceptual lenses such as linear regression formulation (Garg et al., 2022; Li et al., 2023b), meta-learning (Dai et al., 2023; Irie et al., 2022), and Bayesian inference (Xie et al., 2022).”

“Recent works (Dai et al., 2023; Irie et al., 2022) show that ICL can be considered as implicit fine-tuning based on the dual form between Transformer attention and gradient descents.”

“Many works (Aizerman et al., 1964; Irie et al., 2022; Dai et al., 2023; Ren and Liu, 2023) have demonstrated the similarity between the linear attention and the linear layers optimized by gradient descent.”

“(Irie et al., 2022) rewrite the dual form of a linear per-ceptron in terms of query-key-value attention, and use it to analyze how a trained model is affected by its training samples.”

“The idea in this paper to explain language models as meta-optimizers is inspired by Aizerman et al. (1964); Irie et al. (2022).”

“Inspired by the dual form of linear attention proposed in Aiserman et al. [1964] and Irie et al. [2022], the ICL process is interpreted as implicit fine-tuning in the setting of linear attention by Dai et al. [2022].”

“This can allow for learning statistical regularities in the patterns’ superpositions to represent useful prototypes, likely benefiting generalization (Irie et al., 2022).”

“Recently, it has been analyzed that feed-forward layers in transformers mirror memory mechanisms on the keys and values [15, 26].”

“…are capable of either memorizing the knowledge abstraction (e.g., persistent memory) [39], or memorizing the data-dependent information (also known as contextual memory), through recurrence [97, 94, 95], Transformers [110, 32–34, 111, 99], gradient [81, 23], or other learning paradigms [112, 113].”

“Generalizing a classic result on kernel machines 41 , Irie and colleagues 12 showed that this construction is equivalent (in the sense of producing the same output for a given input) to the following linear key-value memory: where (using our earlier notation) α = qK ⊤ with v n = e n , k n = x n ,…”

“A dual form exists between attention and gradient descent-based optimization (Irie et al., 2022a), directly connecting the test input to demonstrations.”

“The terminology of primal and dual follows prior work that has explored similar mathematical formulations outside of TTT [35, 7, 61].”

“Prior works (Dai et al., 2022; Irie et al., 2022) have shown the inherent duality between attention and gradient descent.”

“Based on the dual view that the backward process on a linear neural layer is equivalent to the forward process on a linear attention layer , Irie et al. (2022); Dai et al. (2022) proves the mathematical equivalence, illustrating the implicit gradient descent implementation with a linear attention.”

“…aspects of biological intelligence [Kanerva, 1988, Hawkins and Blakeslee, 2004] and provided novel perspectives on machine learning toolsets like dot-product attention, transformers, and linear layers of deep neural networks [Ramsauer et al., 2020, Bricken and Pehlevan, 2021, Irie et al., 2022].”

“The recommended networks by Hochreiter and Schmid Huber [30] because the last state needed to be sufficiently recent and thus influenced the present state, the RNN model may inaccurately predict the current state [31].”

“Finally, we remark that delta rule is closely related to meta or online learning via gradient descent [70, 38], which has been revisited in recent works like Longhorn [56] and TTT [107].”

“, 2 , Institute Technology Previous studies (Dai et al., 2023; Irie et al., 2022; von Oswald q 1 r 1 q 2 r 2 q q N r N d 1 d 2 d N _ ... q 1 r 1 q 2 r 2 q q N r N ... et al., 2023; Akyürek et al., 2023) have theoretically inferred that ICL can be viewed as an implicit finetuning process, with…”

“We use the multi-head version of the computation above (Irie et al., 2022c) to directly replace the multi-head self-attention layer in the Transformer, yielding a “Recursive Self-Transformer”.”

“In fact, Transformers with linearised attention (Katharopoulos et al., 2020) have a dual form (Aizerman et al., 1964; Irie et al., 2022a) that is an FWP with an outer product-based weight generation mechanism (Schmidhuber, 1991a; 1992).”

“…is the same as the classic derivation in ML that derives the duality between the perception and its Vanilla FWP dual form, kernel machines by Aizerman et al. 110 (1964); based on this parallel, the vanilla FWP computation is the primal form , and (unnormalized) attention is its dual form 111 .”

“As an alternative, studies have approximated the attention mechanism by the linear product (Irie et al., 2022; Bahdanau et al., 2016; Luong et al., 2015).”

“The networks were proposed by Hochreiter and Schmid Huber [1,38].”

“To analyze the robustness and enable in-context learning [11, 20, 34, 87] for VQA models, we utilize the proposed General and Yes/No BQ datasets, which are structured as {Image, MQ, 21 (BQ + corresponding similarity score)}.”

“This adjustment is achieved through the meta-gradient mechanism (Dai et al., 2022; Yang et al., 2023; Irie et al., 2022), meaning the few-shot examples influence the model as if performing implicit instruction tuning.”

“Note that this relation is analogous to the famous duality that connects the perceptron to kernel machines (Aizerman et al., 1964; Irie et al., 2022).”

“…and ICP can be used effectively for data selection, a key component to connecting these ideas lies in a recent body of work which suggests that in-context learning implicitly performs gradient descent by constructing meta-gradients (Irie et al., 2022; Dai et al., 2023; Von Oswald et al., 2023).”

“Variants of FWPs whose weight generation is based on outer products between keys and values [7] have been shown [9] to be equivalent to Linear Transformers [11] (using the mathematical equivalence known from perceptron/kernel machine duality [48, 49]).”

“We adapt the ideas of ICL as Transformer attention mechanism by a dual implicit gradient descent optimization from (Dai et al., 2022; Von Oswald et al., 2023; Irie et al., 2022a).”

“Inspired by the dual form of linear attention proposed in Aiserman et al. [1964] and Irie et al. [2022], the ICL process is interpreted as implicit fine-tuning in the setting of linear attention by Dai et al. [2022].”

“Irie et al. [21] and Dai et al. [10] explain ICL as implicit fine-tuning by figuring out a dual form of gradient descent for linear Transformer attention.”

“From a theoretical standpoint, the groundwork of our in-context learning analysis relies upon the principles of simplified linear attention (Irie et al., 2022; Dai et al., 2023).”

“Note that this relation is analogous to the one that connects the perceptron to kernel machines (Aizerman et al., 1964; Irie et al., 2022a).”

“W K b i is equal to the historical input vectors: In the standard gradient descent with loss L , e i = − η ∂ L ∂y i and η is the learning rate and y i = W i x i is the output of the linear layer using the weight matrix W i at step t (Irie et al., 2022).”

“The residual rows remain worse than their non-residual counterparts in Tables 22 and 23, while SwiGLU requires additional stabilization and still stays below the shallow reference.”

“The theoretical motivation for this approach stems from previous studies [2, 19], which demonstrate that linear layers optimized via gradient descent can be viewed through the lens of linear attention mechanisms.”

“Based Optimization and Attention Irie et al. (2022) present that linear layers optimized by gradient descent have a dual form of linear attention, which motivates us to view k NN-MT as meta-optimizers.”

“Firstly, Transformers with linearised self-attention [9, 10] have a “dual form” that is an outer product-based FWP: exactly the same mathematical relation [11] maps the perceptron to its dual form, the kernel machine [12].”

“Also, recently, the duality of meta-learning perspective and the architecture has been in the core of attention of several studies [58, 63, 92, 93].”

“Online Update From the perspective of fast weight programming (Irie et al., 2022a) and test-time training (Sun et al., 2024a) and regression (Wang et al., 2025), the hidden state S can be interpreted as a (fast) weight matrix, with the delta rule optimizing the online regression objective where β t…”

“A linear layer trained by gradient descent is itself such a memory, written additively, since its weight is the initialization plus a sum of outer products of its inputs and errors and its output is an unnormalized attention read over that history [1, 12].”

“2, but not in the experiments presented here (following Irie et al. (2022b)’s few-shot image classification setting).”

“The theoretical motivation for this approach stems from previous studies [2, 19], which demonstrate that linear layers optimized via gradient descent can be viewed through the lens of linear attention mechanisms.”

“With a scalar step size 𝛽 𝑡 ∈ [0 , 1] , the update is When ‖ 𝑘 𝑡 ‖ 2 = 1 , the matrix 𝑘 𝑡 𝑘 ⊤ 𝑡 is a projector, so 𝛽 𝑡 = 1 overwrites the association at key 𝑘 𝑡 and 𝛽 𝑡 = 0 leaves it unchanged.”

“…learned algorithm (Garg et al., 2022), as gradient descent or ridge regression run by attention (von Oswald et al., 2023; Akyürek et al., 2023; Ma-hankali et al., 2024) and as implicit fine-tuning (Dai et al., 2023), and training, conversely, as attention over training patterns (Irie et al., 2022).”

“2 Background: Dual Form of a Linear Layer Here we briefly review the dual form of a linear layer trained by gradient descent as a linear attention system [2, 1], which is our starting point.”

“Since a gradient step can itself be interpreted as a linear-attention update (Irie et al., 2022), some of the gains of TTT variants may therefore be a result of their larger state capacity, particularly at long context lengths.”

“Complementary studies relate ICL to gradient-based optimization or kernel regression in simplified settings, highlighting the role of attention mechanisms and induction heads in contextual pattern matching (Irie et al., 2022; Akyürek et al., 2022; Olsson et al., 2022).”

“To this end, we adopt the dual-form perspective of linear attention [6,13,25], which reformulates attention as an implicit linear transformation whose weight matrix is constructed from key–value pairs and applied to the query to produce the output.”