CoChat
Sign Up
PreprintCited by 77Topic ModelingNeural Networks and ApplicationsNatural Language Processing Techniques

Linear Transformers Are Secretly Fast Weight Programmers

Open access
Schlag, Imanol, Irie, Kazuki, Schmidhuber, Jürgen·2021·arXiv
DOI 10.48550/arxiv.2102.11174arXiv 2102.11174
Open this paper in CoChat — free accountOpen PDF

Abstract

We show the formal equivalence of linearised self-attention mechanisms and fast weight controllers from the early '90s, where a ``slow" neural net learns by gradient descent to program the ``fast weights" of another net through sequences of elementary programming instructions which are additive outer products of self-invented activation patterns (today called keys and values). Such Fast Weight Programmers (FWPs) learn to manipulate the contents of a finite memory and dynamically interact with it. We infer a memory capacity limitation of recent linearised softmax attention variants, and replace the purely additive outer products by a delta rule-like programming instruction, such that the FWP can more easily learn to correct the current mapping from keys to values. The FWP also learns to compute dynamically changing learning rates. We also propose a new kernel function to linearise attention which balances simplicity and effectiveness. We conduct experiments on synthetic retrieval problems as well as standard machine translation and language modelling tasks which demonstrate the benefits of our methods.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2102.11174

Cites

0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Replacing the Hebbian rule with the Delta rule improves memory management and enhances memory capacity in recurrent neural networks.

    “To improve the memory management and to enhance the memory capacity of the above group, several studies suggest replacing Hebbian rule with Delta rule as the learning algorithm in recurrent neural networks (Schlag et al. 2021), resulting in models such as DeltaNet (Schlag et al. 2021), Longhorn (Liu et al. 2024b), and RWKV7 (Peng et al. 2025b).”

    From Nested Learning: The Illusion of Deep Learning Architectures (2025) · cites this paper as Schlag et al. 2021

    The Introduction (p. 1) says the paper replaces a purely additive memory update with a delta-rule-like instruction to select which associations to keep or delete. Section 6.3 (p. 8) reports improvements for the resulting Delta Network over linear Transformers with the sum update rule. The evidence concerns fast-weight linear Transformers, however, not a general improvement in memory capacity across recurrent neural networks; it does not establish an increase in storage capacity. (LLM, conf=0.84)

    Experimental — accuracy not yet measured

  • Many other modern RNN layers such as linear attention and DeltaNet are also inspired by the idea of FWPs.

    “Many other modern RNN layers such as linear attention [44,63] and DeltaNet [62,83] are also inspired by the idea of FWPs.”

    From Learning to (Learn at Test Time): RNNs with Expressive Hidden States (2024) · cites this paper as [63]

    The Introduction (p. 1) says linear attention is formally equivalent to an FWP and says the authors introduced a delta-rule programming instruction inspired by work on FWPs. The Conclusion (p. 9) likewise presents the FWP perspective as the basis for that instruction. These passages support the DeltaNet part, but they do not establish that linear attention itself was inspired by FWPs rather than subsequently recognized as equivalent. (LLM, conf=0.84)

    Experimental — accuracy not yet measured

  • The paper's baseline, Mamba, is only one of the many recent RNN layers that inherit the linear (matrix) hidden states of linear attention.

    “Our baseline, Mamba [27], is only one of the many recent RNN layers that inherit the linear (matrix) hidden states of linear attention [44,63].”

    From Learning to (Learn at Test Time): RNNs with Expressive Hidden States (2024) · cites this paper as [63]

    Section 3.2 (p. 3) describes a fast-weight matrix in linear attention’s forward computation, supporting the matrix-state part of the claim. The cited paper does not establish that many recent RNN layers inherit those states. The citing sentence calls Mamba the citing authors’ baseline, not this paper’s. (LLM, conf=0.84)

    Experimental — accuracy not yet measured

  • Modern recurrent architectures replace the growing key-value cache with a compact recurrent state, achieving constant cost per token.

    “Modern recurrent architectures approach this challenge by replacing the growing key-value cache with a compact recurrent state [38, 21, 20, 50, 22], achieving constant cost per token.”

    From Learning What to Remember: Test-Time Training via Context Distillation (2026) · cites this paper as [38]

  • Recent variants including DeltaNet, Gated DeltaNet and KDA have narrowed the gap to softmax attention while showing favorable scaling.

    “Recent variants, including DeltaNet [38], Gated DeltaNet [50] and KDA [45], have narrowed the gap to softmax attention while showing favorable scaling.”

    From Learning What to Remember: Test-Time Training via Context Distillation (2026) · cites this paper as [38]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

The publisher hasn’t deposited this paper’s reference list.

Research works printed but not linked: 66

  1. Al-Rfou, R., Choe, D., Constant, N., Guo, M., and Jones, L. Character-level language modeling with deeper selfattention. In Proc. Conference on Artificial Intelligence (AAAI) , pp. 3159-3166, Honolulu, HI, USA, January 2019.
  2. Ba, J., Hinton, G. E., Mnih, V., Leibo, J. Z., and Ionescu, C. Using fast weights to attend to the recent past. In Proc. Advances in Neural Information Processing Systems (NIPS) , pp. 4331-4339, Barcelona, Spain, December 2016.
  3. Baevski, A. and Auli, M. Adaptive input representations for neural language modeling. In Int. Conf. on Learning Representations (ICLR) , New Orleans, LA, USA, May 2019.
  4. Bahdanau, D., Cho, K., and Bengio, Y. Neural machine translation by jointly learning to align and translate. In Int. Conf. on Learning Representations (ICLR) , San Diego, CA, USA, May 2015.
  5. Bello, I. Lambdanetworks: Modeling long-range interactions without attention. In Int. Conf. on Learning Representations (ICLR) , Virtual only, May 2021.
  6. Cheng, J., Dong, L., and Lapata, M. Long short-term memory-networks for machine reading. In Proc. Conf. on Empirical Methods in Natural Language Processing (EMNLP) , pp. 551-561, Austin, TX, USA, November 2016.
  7. Clevert, D.-A., Unterthiner, T., and Hochreiter, S. Fast and accurate deep network learning by exponential linear units (ELUs). In Int. Conf. on Learning Representations (ICLR) , San Juan, Puerto Rico, May 2016.
  8. Dai, Z., Yang, Z., Yang, Y., Cohen, W. W., Carbonell, J., Le, Q. V., and Salakhutdinov, R. Transformer-XL: Attentive language models beyond a fixed-length context. In Proc. Association for Computational Linguistics (ACL) , pp. 2978-2988, Florence, Italy, July 2019.
  9. Demircigil, M., Heusel, J., L¨ owe, M., Upgang, S., and Vermet, F. On a model of associative memory with huge storage capacity. Journal of Statistical Physics , 168(2): 288-299, 2017.
  10. Devlin, J., Chang, M., Lee, K., and Toutanova, K. BERT: pre-training of deep bidirectional transformers for language understanding. In Proc. North American Chapter of the Association for Computational Linguistics on Human Language Technologies (NAACL-HLT) , pp. 41714186, Minneapolis, MN, USA, June 2019.
  11. Feldman, J. A. Dynamic connections in neural networks. Biological cybernetics , 46(1):27-39, 1982.
  12. Galanti, T. and Wolf, L. On the modularity of hypernetworks. In Proc. Advances in Neural Information Processing Systems (NeurIPS) , Virtual only, 2020.
  13. Graves, A. Generating sequences with recurrent neural networks. Preprint arXiv:1308.0850 , 2013. · arXiv
  14. Greff, K., van Steenkiste, S., and Schmidhuber, J. On the binding problem in artificial neural networks. Preprint arXiv:2012.05208 , 2020. · arXiv
  15. Ha, D., Dai, A., and Le, Q. V. Hypernetworks. In Int. Conf. on Learning Representations (ICLR) , Toulon, France, April 2017.
  16. Hanson, S. J. A stochastic version of the delta rule. Physica D: Nonlinear Phenomena , 42(1-3):265-272, 1990.
  17. Hebb, D. O. The organization of behavior: a neuropsycholocigal theory. A Wiley Book in Clinical Psychology , 62:78, 1949.
  18. Hinton, G. E. and Plaut, D. C. Using fast weights to deblur old memories. In Proc. Conf. of Cognitive Science Society , pp. 177-186, Seatle, WA, USA, July 1987.
  19. Hochreiter, S. and Schmidhuber, J. Long short-term memory. Neural computation , 9(8):1735-1780, 1997.
  20. Hopfield, J. J. Neural networks and physical systems with emergent collective computational abilities. Proc. of the national academy of sciences , 79(8):2554-2558, 1982.
  21. Irie, K., Zeyer, A., Schl¨ uter, R., and Ney, H. Language modeling with deep Transformers. In Proc. Interspeech , pp. 3905-3909, Graz, Austria, September 2019a.
  22. Irie, K., Zeyer, A., Schl¨ uter, R., and Ney, H. Training language models for long-span cross-sentence evaluation. In Proc. IEEE Automatic Speech Recog. and Understanding Workshop (ASRU) , pp. 419-426, Sentosa, Singapore, December 2019b.
  23. Irie, K., Gerstenberger, A., Schl¨ uter, R., and Ney, H. How much self-attention do we need? Trading attention for feed-forward layers. In Proc. IEEE Int. Conf. on Acoustics, Speech and Signal Processing (ICASSP) , pp. 61546158, Virtual only, May 2020.
  24. Jia, X., De Brabandere, B., Tuytelaars, T., and Gool, L. V. Dynamic filter networks. In Proc. Advances in Neural Information Processing Systems (NIPS) , pp. 667-675, Barcelona, Spain, 2016.
  25. Katharopoulos, A., Vyas, A., Pappas, N., and Fleuret, F. Transformers are rnns: Fast autoregressive transformers with linear attention. In Proc. Int. Conf. on Machine Learning (ICML) , Virtual only, July 2020.
  26. Kingma, D. P. and Ba, J. Adam: A method for stochastic optimization. Preprint arXiv:1412.6980 , 2014. · arXiv
  27. Kirsch, L. and Schmidhuber, J. Meta learning backpropagation and improving it. NeurIPS Workshop on MetaLearning , 2020.
  28. Klein, B., Wolf, L., and Afek, Y. A dynamic convolutional layer for short rangeweather prediction. In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR) , pp. 4840-4848, Boston, MA, USA, June 2015.
  29. Kohonen, T. Correlation matrix memories. IEEE Transactions on Computers , 21(4):353-359, 1972.
  30. Kosko, B. Bidirectional associative memories. IEEE Transactions on Systems, Man, and Cybernetics , 18(1):49-60, 1988.
  31. Krotov, D. and Hopfield, J. J. Dense associative memory for pattern recognition. In Proc. Advances in Neural Information Processing Systems (NIPS) , pp. 1172-1180, Barcelona, Spain, December 2016.
  32. Lin, Z., Feng, M., Santos, C. N. d., Yu, M., Xiang, B., Zhou, B., and Bengio, Y. A structured self-attentive sentence embedding. In Int. Conf. on Learning Representations (ICLR) , Toulon, France, April 2017.
  33. Little, W. A. The existence of persistent states in the brain. Mathematical biosciences , 19(1-2):101-120, 1974.
  34. Merity, S., Xiong, C., Bradbury, J., and Socher, R. Pointer sentinel mixture models. In Int. Conf. on Learning Representations (ICLR) , Toulon, France, April 2017.
  35. Miconi, T., Stanley, K., and Clune, J. Differentiable plasticity: training plastic neural networks with backpropagation. In Proc. Int. Conf. on Machine Learning (ICML) , pp. 3559-3568, Stockholm, Sweden, July 2018.
  36. Miconi, T., Rawal, A., Clune, J., and Stanley, K. O. Backpropamine: training self-modifying neural networks with differentiable neuromodulated plasticity. In Int. Conf. on Learning Representations (ICLR) , New Orleans, LA, USA, May 2019.
  37. Munkhdalai, T. and Trischler, A. Metalearning with hebbian fast weights. Preprint arXiv:1807.05076 , 2018. · arXiv
  38. Munkhdalai, T. and Yu, H. Meta networks. In Proc. Int. Conf. on Machine Learning (ICML) , pp. 2554-2563, Sydney, Australia, August 2017.
  39. Munkhdalai, T., Sordoni, A., Wang, T., and Trischler, A. Metalearned neural memory. In Proc. Advances in Neural Information Processing Systems (NeurIPS) , pp. 1331013321, Vancouver, Canada, December 2019.
  40. Noh, H., Seo, P. H., and Han, B. Image question answering using convolutional neural network with dynamic parameter prediction. In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR) , pp. 30-38, Las Vegas, NV, USA, 2016.
  41. Ott, M., Edunov, S., Grangier, D., and Auli, M. Scaling neural machine translation. In Proc. Conf. on Machine Translation: Research Papers , pp. 1-9, Brussels, Belgium, October 2018.
  42. Ott, M., Edunov, S., Baevski, A., Fan, A., Gross, S., Ng, N., Grangier, D., and Auli, M. fairseq: A fast, extensible toolkit for sequence modeling. In Proc. North American Chapter of the Association for Computational Linguistics on Human Language Technologies (NAACL-HLT), Demonstrations , pp. 48-53, Minneapolis, MN, USA, June 2019.
  43. Palm, G. On associative memory. Biological cybernetics , 36(1):19-31, 1980.
  44. Papineni, K., Roukos, S., Ward, T., and Zhu, W.-J. Bleu: a method for automatic evaluation of machine translation. In Proc. Association for Computational Linguistics (ACL) , pp. 311-318, Philadelphia, PA, USA, July 2002.
  45. Parikh, A. P., T¨ ackstr¨ om, O., Das, D., and Uszkoreit, J. A decomposable attention model for natural language inference. In Proc. Conf. on Empirical Methods in Natural Language Processing (EMNLP) , pp. 2249-2255, Austin, TX, USA, November 2016.
  46. Paszke, A. et al. Pytorch: An imperative style, highperformance deep learning library. In Proc. Advances in Neural Information Processing Systems (NeurIPS) , pp. 8026-8037, Vancouver, Canada, December 2019.
  47. Peng, H., Pappas, N., Yogatama, D., Schwartz, R., Smith, N. A., and Kong, L. Random feature attention. In Int. Conf. on Learning Representations (ICLR) , Virtual only, 2021.
  48. Perez, E., Strub, F., De Vries, H., Dumoulin, V., and Courville, A. FiLM: Visual reasoning with a general conditioning layer. In Proc. AAAI Conf. on Artificial Intelligence , pp. 3942-3951, New Orleans, LA, USA, February 2018.
  49. Post, M. A call for clarity in reporting BLEU scores. In Proc. Conf. on Machine Translation , pp. 186-191, Brussels, Belgium, October 2018.
  50. Rae, J. W., Potapenko, A., Jayakumar, S. M., Hillier, C., and Lillicrap, T. P. Compressive transformers for longrange sequence modelling. In Int. Conf. on Learning Representations (ICLR) , Virtual only, April 2020.
  51. Ramsauer, H., Sch¨ afl, B., Lehner, J., Seidl, P., Widrich, M., Gruber, L., Holzleitner, M., Adler, T., Kreil, D., Kopp, M. K., Klambauer, G., Brandstetter, J., and Hochreiter, S. Hopfield networks is all you need. In Int. Conf. on Learning Representations (ICLR) , Virtual only, May 2021.
  52. Schlag, I. and Schmidhuber, J. Gated fast weights for onthe-fly neural program generation. In NIPS Metalearning Workshop , Long Beach, CA, USA, December 2017.
  53. Schlag, I. and Schmidhuber, J. Learning to reason with third order tensor products. In Proc. Advances in Neural Information Processing Systems (NIPS) , pp. 9981-9993, Montr´ eal, Canada, December 2018.
  54. Schlag, I., Munkhdalai, T., and Schmidhuber, J. Learning associative inference using fast weight memory. In Int. Conf. on Learning Representations (ICLR) , Virtual only, May 2021.
  55. Schmidhuber, J. Learning to control fast-weight memories: An alternative to dynamic recurrent networks. Neural Computation , 4(1):131-139, 1992.
  56. Schmidhuber, J. Reducing the ratio between learning complexity and number of time varying variables in fully recurrent nets. In International Conference on Artificial Neural Networks (ICANN) , pp. 460-463, Amsterdam, Netherlands, September 1993.
  57. Sennrich, R., Haddow, B., and Birch, A. Neural machine translation of rare words with subword units. In Proc. Association for Computational Linguistics (ACL) , pp. 17151725, Berlin, Germany, August 2016.
  58. Shen, Z., Zhang, M., Zhao, H., Yi, S., and Li, H. Efficient attention: Attention with linear complexities. Preprint arXiv:1812.01243 , 2018. · arXiv
  59. Smolensky, P. Tensor product variable binding and the representation of symbolic structures in connectionist systems. Artificial intelligence , 46(1-2):159-216, 1990.
  60. Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., and Salakhutdinov, R. Dropout: a simple way to prevent neural networks from overfitting. The Journal of Machine Learning Research , 15(1):1929-1958, 2014.
  61. Steinbuch, K. Die lernmatrix. Kybernetik , 1(1):36-45, 1961.
  62. Steinbuch, K. and Piske, U. A. W. Learning matrices and their applications. IEEE Transactions on Electronic Computers , 12(6):846-862, 1963.
  63. Tay, Y., Dehghani, M., Abnar, S., Shen, Y., Bahri, D., Pham, P., Rao, J., Yang, L., Ruder, S., and Metzler, D. Long range arena: A benchmark for efficient transformers. In Int. Conf. on Learning Representations (ICLR) , Virtual only, May 2021.
  64. Tsai, Y.-H. H., Bai, S., Yamada, M., Morency, L.-P., and Salakhutdinov, R. Transformer dissection: An unified understanding for transformer's attention via the lens of kernel. In Proc. Conf. on Empirical Methods in Natural Language Processing (EMNLP) , pp. 4344-4353, Hong Kong, China, November 2019.
  65. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., and Polosukhin, I. Attention is all you need. In Proc. Advances in Neural Information Processing Systems (NIPS) , pp. 5998-6008, Long Beach, CA, USA, December 2017.
  66. Widrow, B. and Hoff, M. E. Adaptive switching circuits. In Proc. IRE WESCON Convention Record , pp. 96-104, Los Angeles, CA, USA, August 1960.

Cited by

Sorted by citations

Related

Authors
Schlag, I., Irie, K., & Schmidhuber, J.
Affiliations
University of Applied Sciences and Arts of Southern Switzerland (CH)
ORCID
Schmidhuber, Jürgen
Venue
arXiv
Publisher
arXiv
Published
2021
Pages
9355-9366
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2102.11174
arXiv
2102.11174
OpenAlex
W3166702123
Cited by
60
References
0
Citations by year
2025: 8 · 2024: 13 · 2023: 15 · 2022: 16 · 2021: 8
Type
posted content
Language
en
Open access
Licence
creativecommons.org/licenses/by/4.0/legalcode
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University) (cc-by)
  • International Conference on Machine Learning
Funders
  • National Science Foundation
  • Nvidia
  • Schweizerischer Nationalfonds zur Förderung der Wissenschaftlichen Forschung (192356, 200021)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Neural Networks and Applications
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
Keywords
memory capacity limits, softmax attention, delta rule, dynamic learning rate, kernel function design, retrieval, machine translation, language modeling

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review