CoChat
Sign Up
PreprintCited by 5Topic ModelingNatural Language Processing TechniquesDomain Adaptation and Few-Shot Learning

Test-time regression: a unifying framework for designing sequence models with associative memory

Open access
Wang, Ke Alexander, Shi, Jiaxin, Fox, Emily B.·2025·arXiv
DOI 10.48550/arxiv.2501.12352arXiv 2501.12352
Open this paper in CoChat — free accountOpen PDF

Abstract

Sequence models lie at the heart of modern deep learning. However, rapid advancements have produced a diversity of seemingly unrelated architectures, such as Transformers and recurrent alternatives. In this paper, we introduce a unifying framework to understand and derive these sequence models, inspired by the empirical importance of associative recall, the capability to retrieve contextually relevant tokens. We formalize associative recall as a two-step process, memorization and retrieval, casting memorization as a regression problem. Layers that combine these two steps perform associative recall via ``test-time regression'' over its input tokens. Prominent layers, including linear attention, state-space models, fast-weight programmers, online learners, and softmax attention, arise as special cases defined by three design choices: the regression weights, the regressor function class, and the test-time optimization algorithm. Our approach clarifies how linear attention fails to capture inter-token correlations and offers a mathematical justification for the empirical effectiveness of query-key normalization in softmax attention. Further, it illuminates unexplored regions within the design space, which we use to derive novel higher-order generalizations of softmax attention. Beyond unification, our work bridges sequence modeling with classic regression methods, a field with extensive literature, paving the way for developing more powerful and theoretically principled architectures.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2501.12352

Cites

5 of 5 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Miras can distinguish models with Hebbian-like update (dot product similarity) and delta rule update (regression loss), contrary to Wang et al. (2025).

    “For example, contrary to Wang et al. (2025), Miras can distinguish models with Hebbian-like update (with dot product similarity) and delta rule update (with regression loss).”

    From It's All Connected: A Journey Through Test-Time Memorization, Attentional Bias, Retention, and Online Optimization (2025) · cites this paper as Wang et al. (2025)

    Wang et al. distinguish the two updates rather than treating them as indistinguishable. The “Linear attention” section gives its additive key–value update (Eq. 2), while “Vignette 1: Linear attention is (suboptimal) linear least squares regression” derives DeltaNet’s error-correcting update from single-example SGD (Eqs. 27–29). The claim about what Miras can do is the citing authors’ comparison, not a finding attributed to Wang et al. (LLM, conf=0.85)

    Checked by a reasoning model — may be wrong.

    Experimental — accuracy not yet measured

  • Attention and its more expressive variants are instances of Definition 1 when the optimal non-parametric solution to the mapping is used.

    “Therefore, attention and its more expressive variants (Wang et al. 2025) also are instances of Definition 1, when instead of gradient descent or other parametric methods, we find the optimal non-parametric solution to the mapping.”

    From Nested Learning: The Illusion of Deep Learning Architectures (2025) · cites this paper as Wang et al. 2025

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

  • Convolutional Sequence to Sequence Learning

    Gehring, J., Auli, M., Grangier, D., Yarats, D., & Dauphin, Y. N. (2017) · arXiv · doi:10.48550/arxiv.1705.03122 · Cited by 2,850 · cited 1× · see citing works

    “…state-space models with more general structured masks (Dao and Gu, 2024), convolutional architectures (Shi et al., 2023; Poli et al., 2023; Gehring et al., 2017; Gu et al., 2021; Bradbury et al., 2017; Smith et al., 2023; Hochreiter and Schmidhuber, 1997), and more novel computational…”

  • Scaling Laws for Neural Language Models

    Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., & Amodei, D. (2020) · arXiv · Cited by 1,749 · cited 1× · see citing works

    “…approaches will depend heavily on efficient hardware implementations (Hua et al., 2022; Yang et al., 2024c; Sun et al., 2024; Dao and Gu, 2024; Dao et al., 2022; Oswald et al., 2025), aligning with established compute-scaling principles (Sutton, 2019; Hoffmann et al., 2022; Kaplan et al., 2020).”

  • Rethinking Attention with Performers

    Choromanski, K., Likhosherstov, V., Dohan, D., Song, X., Gane, A., Sarlos, T., Hawkins, P., Davis, J., Mohiuddin, A., Kaiser, L., et al. (2020) · arXiv · doi:10.48550/arxiv.2009.14794 · Cited by 695 · cited 2× · see citing works

    “This relationship with kernel regression has been used by many previous works to derive more efficient versions of self-attention (Peng et al., 2020; Chen et al., 2021; Choromanski et al., 2020; Lu et al., 2021; Chen et al., 2024), borrowing techniques from kernel approximation methods.”

  • Memory Mosaics

    Zhang, J., Nolte, N., Sadhukhan, R., Chen, B., & Bottou, L. (2024) · arXiv · doi:10.48550/arxiv.2405.06394

  • MesaNet: Sequence Modeling by Locally Optimal Test-Time Training

    von Oswald, J., Scherrer, N., Kobayashi, S., Versari, L., Yang, S., Mittal, S., Schlegel, M., Maile, K., Schimpf, Y., Sieberling, O., et al. (2025) · arXiv · cited 2×

    “Developing hardware-aware optimizations to compute Equation 52 similar to FlashAttention (Dao et al., 2022) and MesaNet (Oswald et al., 2025) is a promising future direction.”

Flagged first, then by citations

Research works printed but not linked: 117

  1. K. Ahn, X. Cheng, H. Daneshmand, and S. Sra. Transformers learn to implement preconditioned gradient descent for in-context learning. Advances in Neural Information Processing Systems, 36: 45614-45650, Dec. 2023. URL https://proceedings.neurips.cc/paper_files/paper/2023/hash/ 8ed3d610ea4b68e7afb30ea7d01422c6-Abstract-Conference.html.
  2. J. Ainslie, J. Lee-Thorp, M. d. Jong, Y. Zemlyanskiy, F. Lebron, and S. Sanghai. GQA: Training Generalized Multi- Query Transformer Models from Multi-Head Checkpoints. In The 2023 Conference on Empirical Methods in Natural Language Processing, Dec. 2023. URL https://openreview.net/forum?id=hmOwOZWzYE.
  3. Y. Aksenov, N. Balagansky, S. M. L. C. Vaina, B. Shaposhnikov, A. Gorbatovski, and D. Gavrilov. Linear Transformers with Learnable Kernel Functions are Better In-Context Models, Feb. 2024. URL http://arxiv.org/abs/2402. 10644. arXiv:2402.10644 [cs] version: 1. · arXiv
  4. E. Akyürek, M. Damani, L. Qiu, H. Guo, Y. Kim, and J. Andreas. The Surprising Effectiveness of Test-Time Training for Abstract Reasoning, Nov. 2024. URL http://arxiv.org/abs/2411.07279. arXiv:2411.07279 [cs] version: 1. · arXiv
  5. A. F. Ansari, L. Stella, A. C. Turkmen, X. Zhang, P. Mercado, H. Shen, O. Shchur, S. S. Rangapuram, S. P. Arango, S. Kapoor, J. Zschiegner, D. C. Maddix, H. Wang, M. W. Mahoney, K. Torkkola, A. G. Wilson, M. Bohlke-Schneider, and B. Wang. Chronos: Learning the language of time series. Transactions on Machine Learning Research, 2024. ISSN 2835-8856. URL https://openreview.net/forum?id=gerNCVqqtR.
  6. S. Arora, S. Eyuboglu, A. Timalsina, I. Johnson, M. Poli, J. Zou, A. Rudra, and C. Re. Zoology: Measuring and Im- proving Recall in Efficient Language Models. In The Twelfth International Conference on Learning Representations, Oct. 2023a. URL https://openreview.net/forum?id=LY3ukUANko.
  7. S. Arora, B. Yang, S. Eyuboglu, A. Narayan, A. Hojel, I. Trummer, and C. Ré. Language Models Enable Simple Systems for Generating Structured Views of Heterogeneous Data Lakes, Apr. 2023b. URL http://arxiv.org/abs/2304. 09433. arXiv:2304.09433 [cs. · arXiv
  8. S. Arora, S. Eyuboglu, M. Zhang, A. Timalsina, S. Alberti, J. Zou, A. Rudra, and C. Re. Simple linear attention language models balance the recall-throughput tradeoff. In Proceedings of the 41st International Conference on Machine Learning, pages 1763-1840. PMLR, July 2024. URL https://proceedings.mlr.press/v235/arora24a.html. ISSN: 2640-3498.
  9. M. Beck, K. Pöppel, M. Spanring, A. Auer, O. Prudnikova, M. Kopp, G. Klambauer, J. Brandstetter, and S. Hochre- iter. xLSTM: Extended Long Short-Term Memory, May 2024. URL http://arxiv.org/abs/2405.04517. arXiv:2405.04517 [cs, stat]. · arXiv
  10. A. Behrouz, P. Zhong, and V. Mirrokni. Titans: Learning to Memorize at Test Time, Dec. 2024. URL http://arxiv. org/abs/2501.00663. arXiv:2501.00663 [cs. · arXiv
  11. G. Bertasius, H. Wang, and L. Torresani. Is Space-Time Attention All You Need for Video Understanding? In Proceedings of the 38th International Conference on Machine Learning, pages 813-824. PMLR, July 2021. URL https: //proceedings.mlr.press/v139/bertasius21a.html. ISSN: 2640-3498.
  12. A. Bietti, V. Cabannes, D. Bouchacourt, H. Jegou, and L. Bottou. Birth of a Transformer: A Memory Viewpoint. In Thirty-seventh Conference on Neural Information Processing Systems, Nov. 2023. URL https://openreview.net/ forum?id=3X2EbBLNsk.
  13. S. Boyd and L. Vandenberghe. Convex optimization. Cambridge University Press, Cambridge, 2004.
  14. J. Bradbury, S. Merity, C. Xiong, and R. Socher. Quasi-recurrent neural networks. In International conference on learning representations, 2017. URL https://openreview.net/forum?id=H1zJ-v5xl.
  15. T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei. Language Models are Few-Shot Learners, July 2020. URL http://arxiv.org/abs/2005.14165. arXiv:2005.14165 [cs. · arXiv
  16. V. Cabannes, E. Dohmatob, and A. Bietti. Scaling Laws for Associative Memories. In The Twelfth International Conference on Learning Representations, Oct. 2023. URL https://openreview.net/forum?id=Tzh6xAJSll.
  17. F. T. Castoldi and M. L. R. de Campos. Minimum-disturbance description for the development of adaptation algo- rithms and a new leakage least squares algorithm. In 2009 IEEE International Conference on Acoustics, Speech and Signal Processing, pages 3129-3132, Apr. 2009. doi: 10.1109/ICASSP.2009.4960287. URL https://ieeexplore. ieee.org/document/4960287. ISSN: 2379-190X. · DOI
  18. N. Cesa-Bianchi, A. Conconi, and C. Gentile. On the generalization ability of on-line learning algorithms. IEEE Trans- actions on Information Theory, 50(9):2050-2057, Sept. 2004. ISSN 1557-9654. doi: 10.1109/TIT.2004.833339. URL https://ieeexplore.ieee.org/document/1327806/?arnumber=1327806. Conference Name: IEEE Transac- tions on Information Theory. · DOI
  19. H. Chen, Liuzhicheng, X. Wang, Y. Tian, and Y. Wang. DiJiang: Efficient Large Language Models through Compact Kernelization. In Forty-first International Conference on Machine Learning, June 2024. URL https://openreview.net/forum?id=0uUHfhXdnH&referrer=%5Bthe%20profile%20of%20Yuchuan%20Tian% 5D(%2Fprofile%3Fid%3D~Yuchuan_Tian1).
  20. Y. Chen, Q. Zeng, H. Ji, and Y. Yang. Skyformer: remodel self-attention with Gaussian kernel and nyström method. In Proceedings of the 35th International Conference on Neural Information Processing Systems, NIPS '21, pages 2122- 2135, Red Hook, NY, USA, 2021. Curran Associates Inc. ISBN 978-1-7138-4539-3.
  21. K. Clark, K. Guu, M.-W. Chang, P. Pasupat, G. Hinton, and M. Norouzi. Meta-Learning Fast Weight Language Models. In Y. Goldberg, Z. Kozareva, and Y. Zhang, editors, Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 9751-9757, Abu Dhabi, United Arab Emirates, Dec. 2022. Association for Computational Linguistics. doi: 10.18653/v1/2022.emnlp-main.661. URL https://aclanthology.org/2022. emnlp-main.661. · DOI
  22. T. Dao and A. Gu. Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality. In Forty-first International Conference on Machine Learning, June 2024.
  23. T. Dao, D. Y. Fu, S. Ermon, A. Rudra, and C. Re. FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness. In Advances in Neural Information Processing Systems, Oct. 2022. URL https://openreview.net/ forum?id=H4DqfPSibmx.
  24. S. De, S. L. Smith, A. Fernando, A. Botev, G. Cristian-Muraru, A. Gu, R. Haroun, L. Berrada, Y. Chen, S. Srinivasan, G. Desjardins, A. Doucet, D. Budden, Y. W. Teh, R. Pascanu, N. D. Freitas, and C. Gulcehre. Griffin: Mixing Gated Linear Recurrences with Local Attention for Efficient Language Models, Feb. 2024. URL http://arxiv.org/abs/ 2402.19427. arXiv:2402.19427 [cs. · arXiv
  25. M. Dehghani, J. Djolonga, B. Mustafa, P. Padlewski, J. Heek, J. Gilmer, A. P. Steiner, M. Caron, R. Geirhos, I. Alabdul- mohsin, R. Jenatton, L. Beyer, M. Tschannen, A. Arnab, X. Wang, C. R. Ruiz, M. Minderer, J. Puigcerver, U. Evci, M. Kumar, S. V. Steenkiste, G. F. Elsayed, A. Mahendran, F. Yu, A. Oliver, F. Huot, J. Bastings, M. Collier, A. A. Gritsenko, V. Birodkar, C. N. Vasconcelos, Y. Tay, T. Mensink, A. Kolesnikov, F. Pavetic, D. Tran, T. Kipf, M. Lucic, X. Zhai, D. Keysers, J. J. Harmsen, and N. Houlsby. Scaling Vision Transformers to 22 Billion Parameters. In Proceedings of the 40th International Conference on Machine Learning, pages 7480-7512. PMLR, July 2023. URL https://proceedings.mlr.press/v202/dehghani23a.html. ISSN: 2640-3498.
  26. J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. BERT: Pre-training of Deep Bidirectional Transformers for Lan- guage Understanding. In J. Burstein, C. Doran, and T. Solorio, editors, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10.18653/v1/N19-1423. URL https://aclanthology.org/N19-1423/. · DOI · Paper page
  27. Q. Dong, L. Li, D. Dai, C. Zheng, J. Ma, R. Li, H. Xia, J. Xu, Z. Wu, T. Liu, B. Chang, X. Sun, L. Li, and Z. Sui. A Survey on In-context Learning, Oct. 2024. URL http://arxiv.org/abs/2301.00234. arXiv:2301.00234 [cs. · arXiv
  28. A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International conference on learning representations, 2021. URL https://openreview. net/forum?id=YicbFdNTTy.
  29. J. Duchi, E. Hazan, and Y. Singer. Adaptive subgradient methods for online learning and stochastic optimization. Journal of Machine Learning Research, 12(61):2121-2159, 2011. URL http://jmlr.org/papers/v12/duchi11a. html.
  30. J. Fan. Local Polynomial Modelling and Its Applications: Monographs on Statistics and Applied Probability 66. Routledge, New York, May 2018. ISBN 978-0-203-74872-5. doi: 10.1201/9780203748725. · DOI
  31. D. Y. Fu, T. Dao, K. K. Saab, A. W. Thomas, A. Rudra, and C. Re. Hungry Hungry Hippos: Towards Language Modeling with State Space Models. In The Eleventh International Conference on Learning Representations, Sept. 2022. URL https://openreview.net/forum?id=COZDy0WYGg.
  32. S. Garg, D. Tsipras, P. Liang, and G. Valiant. What can transformers learn in-context? A case study of simple function classes. In A. H. Oh, A. Agarwal, D. Belgrave, and K. Cho, editors, Advances in neural information processing systems, 2022. URL https://openreview.net/forum?id=flNZJ2eOet.
  33. M. Garnelo and W. M. Czarnecki. Exploring the Space of Key-Value-Query Models with Intention, May 2023. URL http://arxiv.org/abs/2305.10203. arXiv:2305.10203. · arXiv
  34. G. Goel and P. Bartlett. Can a transformer represent a Kalman filter? In A. Abate, M. Cannon, K. Margellos, and A. Papachristodoulou, editors, Proceedings of the 6th annual learning for dynamics & control confer- ence, volume 242 of Proceedings of machine learning research, pages 1502-1512. PMLR, July 2024. URL https: //proceedings.mlr.press/v242/goel24a.html.
  35. K. Greff, R. K. Srivastava, J. Koutník, B. R. Steunebrink, and J. Schmidhuber. LSTM: A Search Space Odyssey. IEEE Transactions on Neural Networks and Learning Systems, 28(10):2222-2232, Oct. 2017. ISSN 2162-237X, 2162-2388. doi: 10.1109/TNNLS.2016.2582924. URL http://arxiv.org/abs/1503.04069. arXiv:1503.04069 [cs. · DOI · Paper page
  36. N. Gruver, M. Finzi, S. Qiu, and A. G. Wilson. Large Language Models Are Zero-Shot Time Series Forecasters. Advances in Neural Information Processing Systems, 36:19622- 19635, Dec. 2023. URL https://proceedings.neurips.cc/paper_files/paper/2023/hash/ 3eb7ca52e8207697361b2c0fb3926511-Abstract-Conference.html.
  37. A. Gu and T. Dao. Mamba: Linear-Time Sequence Modeling with Selective State Spaces. In First Conference on Language Modeling, Aug. 2024. URL https://openreview.net/forum?id=tEYskw1VY2#discussion.
  38. A. Gu, T. Dao, S. Ermon, A. Rudra, and C. Ré. HiPPO: Recurrent Memory with Optimal Polynomial Projections. In Ad- vances in Neural Information Processing Systems, volume 33, pages 1474-1487.
  39. Curran Associates, Inc., 2020. URL https://proceedings.neurips.cc/paper/2020/hash/102f0bb6efb3a6128a3c750dd16729be-Abstract. html.
  40. A. Gu, K. Goel, and C. Re. Efficiently Modeling Long Sequences with Structured State Spaces. In International Conference on Learning Representations, Oct. 2021. URL https://openreview.net/forum?id=uYLFoz1vlAC.
  41. A. Gu, A. Gupta, K. Goel, and C. Ré. On the parameterization and initialization of diagonal state space models. In Proceedings of the 36th International Conference on Neural Information Processing Systems, NIPS '22, pages 35971- 35983, Red Hook, NY, USA, 2022.
  42. D. Han, Z. Wang, Z. Xia, Y. Han, Y. Pu, C. Ge, J. Song, S. Song, B. Zheng, and G. Huang. Demystify Mamba in Vision: A Linear Attention Perspective, May 2024. URL https://arxiv.org/abs/2405.16605v1. · arXiv
  43. S. S. Haykin. Adaptive Filter Theory. Pearson, 2014. ISBN 978-0-13-267145-3. Google-Books-ID: J4GRKQEACAAJ.
  44. G. E. Hinton and J. A. Anderson. Parallel Models of Associative Memory. Psychology Press, 1989.
  45. S. Hochreiter and J. Schmidhuber. Long Short-Term Memory. Neural Computation, 9(8):1735-1780, Nov. 1997. ISSN 0899-7667. doi: 10.1162/neco.1997.9.8.1735. URL https://ieeexplore.ieee.org/abstract/document/ 6795963. Conference Name: Neural Computation. · DOI · Paper page
  46. J. Hoffmann, S. Borgeaud, A. Mensch, E. Buchatskaya, T. Cai, E. Rutherford, D. d. L. Casas, L. A. Hendricks, J. Welbl, A. Clark, T. Hennigan, E. Noland, K. Millican, G. v. d. Driessche, B. Damoc, A. Guy, S. Osindero, K. Simonyan, E. Elsen, J. W. Rae, O. Vinyals, and L. Sifre. Training Compute-Optimal Large Language Models, Mar. 2022. URL http://arxiv.org/abs/2203.15556. arXiv:2203.15556 [cs. · arXiv
  47. J. J. Hopfield. Neural networks and physical systems with emergent collective computational abilities. Proceedings of the National Academy of Sciences, 79(8):2554-2558, Apr. 1982. doi: 10.1073/pnas.79.8.2554. URL https://www. pnas.org/doi/abs/10.1073/pnas.79.8.2554. Publisher: Proceedings of the National Academy of Sciences. · DOI · Paper page
  48. W. Hua, Z. Dai, H. Liu, and Q. Le. Transformer Quality in Linear Time. In Proceedings of the 39th International Conference on Machine Learning, pages 9099-9117. PMLR, June 2022. URL https://proceedings.mlr.press/ v162/hua22a.html. ISSN: 2640-3498.
  49. R. M. Johnstone, C. R. Johnson, R. R. Bitmead, and B. D. O. Anderson. Exponential convergence of recursive least squares with exponential forgetting factor. In 1982 21st IEEE Conference on Decision and Control, pages 994-997, Dec. 1982. doi: 10.1109/CDC.1982.268295. URL https://ieeexplore.ieee.org/document/4047398. · DOI
  50. J. Jumper, R. Evans, A. Pritzel, T. Green, M. Figurnov, O. Ronneberger, K. Tunyasuvunakool, R. Bates, A. Žídek, A. Potapenko, A. Bridgland, C. Meyer, S. A. A. Kohl, A. J. Ballard, A. Cowie, B. Romera-Paredes, S. Nikolov, R. Jain, J. Adler, T. Back, S. Petersen, D. Reiman, E. Clancy, M. Zielinski, M. Steinegger, M. Pacholska, T. Berghammer, S. Bodenstein, D. Silver, O. Vinyals, A. W. Senior, K. Kavukcuoglu, P. Kohli, and D. Hassabis. Highly accurate protein structure prediction with AlphaFold. Nature, 596(7873):583-589, Aug. 2021. ISSN 1476-4687. doi: 10. 1038/s41586-021-03819-2. URL https://www.nature.com/articles/s41586-021-03819-2. Publisher: Nature Publishing Group. · DOI · Paper page
  51. J. Kasai, H. Peng, Y. Zhang, D. Yogatama, G. Ilharco, N. Pappas, Y. Mao, W. Chen, and N. A. Smith. Finetuning Pretrained Transformers into RNNs. In M.-F. Moens, X. Huang, L. Specia, and S. W.-t. Yih, editors, Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 10630-10643, Online and Punta Cana, Dominican Republic, Nov. 2021. Association for Computational Linguistics. doi: 10.18653/v1/2021.emnlp-main. 830. URL https://aclanthology.org/2021.emnlp-main.830. · DOI
  52. A. Katharopoulos, A. Vyas, N. Pappas, and F. Fleuret. Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention. In Proceedings of the 37th International Conference on Machine Learning, pages 5156-5165.
  53. PMLR, Nov. 2020. URL https://proceedings.mlr.press/v119/katharopoulos20a.html. ISSN: 2640-3498.
  54. T. Katsch. GateLoop: Fully Data-Controlled Linear Recurrence for Sequence Modeling, Jan. 2024. URL http:// arxiv.org/abs/2311.01927. arXiv:2311.01927 [cs. · arXiv
  55. D. P. Kingma and J. Ba. Adam: A Method for Stochastic Optimization. 2015. URL https://arxiv.org/abs/1412. 6980.
  56. T. Kohonen. Correlation Matrix Memories. IEEE Transactions on Computers, C-21(4):353-359, Apr. 1972. ISSN 1557-9956. doi: 10.1109/TC.1972.5008975. URL https://ieeexplore.ieee.org/abstract/document/5008975. Conference Name: IEEE Transactions on Computers. · DOI · Paper page
  57. T. Kohonen. Self-Organization and Associative Memory, volume 8 of Springer Series in Information Sciences. Springer, Berlin, Heidelberg, 1989. ISBN 978-3-540-51387-2 978-3-642-88163-3. doi: 10.1007/978-3-642-88163-3. URL http: //link.springer.com/10.1007/978-3-642-88163-3. · DOI · Paper page
  58. D. Krotov and J. Hopfield. Large Associative Memory Problem in Neurobiology and Machine Learning, Apr. 2021. URL http://arxiv.org/abs/2008.06996. arXiv:2008.06996. · arXiv
  59. D. Krotov and J. J. Hopfield. Dense Associative Memory for Pattern Recognition. In Advances in Neural Information Processing Systems, volume 29.
  60. Curran Associates, Inc., 2016. URL https://proceedings.neurips.cc/paper_ files/paper/2016/hash/eaae339c4d89fc102edd9dbdb6a28915-Abstract.html.
  61. B. Liu, R. Wang, L. Wu, Y. Feng, P. Stone, and Q. Liu. Longhorn: State Space Models are Amortized Online Learners, Oct. 2024. URL http://arxiv.org/abs/2407.14207. arXiv:2407.14207. · arXiv
  62. I. Loshchilov and F. Hutter. Decoupled weight decay regularization. In International conference on learning represen- tations, 2019. URL https://openreview.net/forum?id=Bkg6RiCqY7.
  63. J. Lu, J. Yao, J. Zhang, X. Zhu, H. Xu, W. Gao, C. XU, T. Xiang, and L. Zhang. SOFT: Softmax-free Trans- former with Linear Complexity. In Advances in Neural Information Processing Systems, volume 34, pages 21297-21309.
  64. Curran Associates, Inc., 2021. URL https://proceedings.neurips.cc/paper/2021/hash/ b1d10e7bafa4421218a51b1e1f1b0ba2-Abstract.html.
  65. H. Mehta, A. Gupta, A. Cutkosky, and B. Neyshabur. Long range language modeling via gated state spaces. In The eleventh international conference on learning representations, 2023. URL https://openreview.net/forum?id= 5MkYIYCbva.
  66. B. Millidge, T. Salvatori, Y. Song, T. Lukasiewicz, and R. Bogacz. Universal Hopfield Networks: A General Framework for Single-Shot Associative Memory Models. In Proceedings of the 39th International Conference on Machine Learn- ing, pages 15561-15583. PMLR, June 2022. URL https://proceedings.mlr.press/v162/millidge22a.html. ISSN: 2640-3498.
  67. E. Nguyen, M. Poli, M. G. Durrant, B. Kang, D. Katrekar, D. B. Li, L. J. Bartie, A. W. Thomas, S. H. King, G. Brixi, J. Sullivan, M. Y. Ng, A. Lewis, A. Lou, S. Ermon, S. A. Baccus, T. Hernandez-Boussard, C. Ré, P. D. Hsu, and B. L. Hie. Sequence modeling and design from molecular to genome scale with Evo. Science, 386(6723):eado9336, Nov. 2024. doi: 10.1126/science.ado9336. URL https://www.science.org/doi/10.1126/science.ado9336. Publisher: American Association for the Advancement of Science. · DOI · Paper page
  68. E. Nichani, J. D. Lee, and A. Bietti. Understanding Factual Recall in Transformers via Associative Memories, Dec. 2024. URL http://arxiv.org/abs/2412.06538. arXiv:2412.06538 [cs. · arXiv
  69. C. Olsson, N. Elhage, N. Nanda, N. Joseph, N. DasSarma, T. Henighan, B. Mann, A. Askell, Y. Bai, A. Chen, T. Conerly, D. Drain, D. Ganguli, Z. Hatfield-Dodds, D. Hernandez, S. Johnston, A. Jones, J. Kernion, L. Lovitt, K. Ndousse, D. Amodei, T. Brown, J. Clark, J. Kaplan, S. McCandlish, and C. Olah. In-context Learning and Induction Heads, Sept. 2022. URL http://arxiv.org/abs/2209.11895. arXiv:2209.11895 [cs]. · arXiv
  70. A. Orvieto, S. L. Smith, A. Gu, A. Fernando, C. Gulcehre, R. Pascanu, and S. De. Resurrecting Recurrent Neural Networks for Long Sequences. In Proceedings of the 40th International Conference on Machine Learning, pages 26670-26698. PMLR, July 2023. URL https://proceedings.mlr.press/v202/orvieto23a.html. ISSN: 2640- 3498.
  71. J. V. Oswald, E. Niklasson, E. Randazzo, J. Sacramento, A. Mordvintsev, A. Zhmoginov, and M. Vladymyrov. Trans- formers Learn In-Context by Gradient Descent. In Proceedings of the 40th International Conference on Machine Learning, pages 35151-35174. PMLR, July 2023. URL https://proceedings.mlr.press/v202/von-oswald23a. html. ISSN: 2640-3498.
  72. B. Peng, D. Goldstein, Q. Anthony, A. Albalak, E. Alcaide, S. Biderman, E. Cheah, X. Du, T. Ferdinan, H. Hou, P. Kazienko, K. K. GV, J. Kocoń, B. Koptyra, S. Krishna, R. M. Jr, J. Lin, N. Muennighoff, F. Obeid, A. Saito, G. Song, H. Tu, C. Wirawan, S. Woźniak, R. Zhang, B. Zhao, Q. Zhao, P. Zhou, J. Zhu, and R.-J. Zhu. Eagle and Finch: RWKV with Matrix-Valued States and Dynamic Recurrence, Sept. 2024. URL http://arxiv.org/abs/2404.05892. arXiv:2404.05892. · arXiv
  73. H. Peng, N. Pappas, D. Yogatama, R. Schwartz, N. Smith, and L. Kong. Random Feature Attention. In International Conference on Learning Representations, Oct. 2020. URL https://openreview.net/forum?id=QtTKTdVrFBB.
  74. T. Poggio. On optimal nonlinear associative recall. Biological Cybernetics, 19(4):201-209, Sept. 1975. ISSN 1432-0770. doi: 10.1007/BF02281970. URL https://doi.org/10.1007/BF02281970. · DOI
  75. M. Poli, S. Massaroli, E. Nguyen, D. Y. Fu, T. Dao, S. Baccus, Y. Bengio, S. Ermon, and C. Ré. Hyena hierarchy: towards larger convolutional language models. In Proceedings of the 40th International Conference on Machine Learning, volume 202 of ICML'23, pages 28043-28078, Honolulu, Hawaii, USA, July 2023. JMLR.org.
  76. Z. Qin, W. Sun, H. Deng, D. Li, Y. Wei, B. Lv, J. Yan, L. Kong, and Y. Zhong. cosFormer: Rethinking Softmax In Attention. In International Conference on Learning Representations, Oct. 2021. URL https://openreview.net/ forum?id=Bl8CQrx2Up4.
  77. Z. Qin, X. Han, W. Sun, D. Li, L. Kong, N. Barnes, and Y. Zhong. The Devil in Linear Transformer. In Y. Goldberg, Z. Kozareva, and Y. Zhang, editors, Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 7025-7041, Abu Dhabi, United Arab Emirates, Dec. 2022. Association for Computational Lin- guistics. doi: 10.18653/v1/2022.emnlp-main.473. URL https://aclanthology.org/2022.emnlp-main.473. · DOI
  78. Z. Qin, X. Han, W. Sun, B. He, D. Li, D. Li, Y. Dai, L. Kong, and Y. Zhong. Toeplitz neural network for sequence modeling. In The eleventh international conference on learning representations, 2023. URL https://openreview. net/forum?id=IxmWsm4xrua.
  79. Z. Qin, S. Yang, W. Sun, X. Shen, D. Li, W. Sun, and Y. Zhong. HGRN2: Gated Linear RNNs with State Expansion. In First Conference on Language Modeling, Aug. 2024. URL https://openreview.net/forum?id=y6SqbJfCSk# discussion.
  80. H. Ramsauer, B. Schäfl, J. Lehner, P. Seidl, M. Widrich, L. Gruber, M. Holzleitner, T. Adler, D. Kreil, M. K. Kopp, G. Klambauer, J. Brandstetter, and S. Hochreiter. Hopfield Networks is All You Need. In International Conference on Learning Representations, 2021. URL https://openreview.net/forum?id=tL89RnzIiCd.
  81. L. Ren, Y. Liu, S. Wang, Y. Xu, C. Zhu, and C. Zhai. Sparse modular activation for efficient sequence modeling. In Thirty-seventh conference on neural information processing systems, 2023. URL https://openreview.net/forum? id=TfbzX6I14i.
  82. R. Ren, Z. Li, and Y. Liu. Can Mamba Always Enjoy the "Free Lunch"?, Oct. 2024. URL http://arxiv.org/abs/ 2410.03810. arXiv:2410.03810 [cs. · arXiv
  83. D. Salinas, V. Flunkert, J. Gasthaus, and T. Januschowski. DeepAR: Probabilistic forecasting with autoregressive recurrent networks. International Journal of Forecasting, 36(3):1181-1191, July 2020. ISSN 0169-2070. doi: 10.1016/ j.ijforecast.2019.07.001. URL https://www.sciencedirect.com/science/article/pii/S0169207019301888. · DOI
  84. Schlag, K. Irie, and J. Schmidhuber. Linear Transformers Are Secretly Fast Weight Programmers. In Proceedings of the 38th International Conference on Machine Learning, pages 9355-9366. PMLR, July 2021. URL https:// proceedings.mlr.press/v139/schlag21a.html. ISSN: 2640-3498.
  85. J. Schmidhuber. Learning to Control Fast-Weight Memories: An Alternative to Dynamic Recurrent Networks. Neural Computation, 4(1):131-139, Jan. 1992. ISSN 0899-7667. doi: 10.1162/neco.1992.4.1.131. URL https://doi.org/ 10.1162/neco.1992.4.1.131. · DOI · Paper page
  86. B. Schölkopf and A. J. Smola. Learning with Kernels: Support Vector Machines, Regularization, Optimization, and Beyond. The MIT Press, Dec. 2001. ISBN 978-0-262-25693-3. doi: 10.7551/mitpress/4175.001.0001. URL https: //direct.mit.edu/books/monograph/1821/Learning-with-KernelsSupport-Vector-Machines. · DOI
  87. N. Shazeer. Fast Transformer Decoding: One Write-Head is All You Need, Nov. 2019. URL http://arxiv.org/abs/ 1911.02150. arXiv:1911.02150 [cs. · arXiv
  88. J. Shi, K. A. Wang, and E. B. Fox. Sequence modeling with multiresolution convolutional memory. In Proceedings of the 40th International Conference on Machine Learning, volume 202 of ICML'23, pages 31312-31327, Honolulu, Hawaii, USA, July 2023. JMLR.org.
  89. J. Siems, T. Carstensen, A. Zela, F. Hutter, M. Pontil, and R. Grazzi. DeltaProduct: Increasing the Expressiv- ity of DeltaNet Through Products of Householders, Feb. 2025. URL http://arxiv.org/abs/2502.10297. arXiv:2502.10297 [cs. · arXiv
  90. J. T. Smith, A. Warrington, and S. Linderman. Simplified state space layers for sequence modeling. In The eleventh in- ternational conference on learning representations, 2023. URL https://openreview.net/forum?id=Ai8Hw3AXqks.
  91. S. Sukhbaatar, E. Grave, G. Lample, H. Jegou, and A. Joulin. Augmenting Self-attention with Persistent Memory, July 2019. URL http://arxiv.org/abs/1907.01470. arXiv:1907.01470. · arXiv
  92. Y. Sun, X. Wang, Z. Liu, J. Miller, A. Efros, and M. Hardt. Test-Time Training with Self-Supervision for Generalization under Distribution Shifts. In Proceedings of the 37th International Conference on Machine Learning, pages 9229- 9248. PMLR, Nov. 2020. URL https://proceedings.mlr.press/v119/sun20b.html. ISSN: 2640-3498.
  93. Y. Sun, L. Dong, S. Huang, S. Ma, Y. Xia, J. Xue, J. Wang, and F. Wei. Retentive Network: A Successor to Transformer for Large Language Models, Aug. 2023. URL http://arxiv.org/abs/2307.08621. arXiv:2307.08621 [cs. · arXiv
  94. Y. Sun, X. Li, K. Dalal, J. Xu, A. Vikram, G. Zhang, Y. Dubois, X. Chen, X. Wang, S. Koyejo, T. Hashimoto, and C. Guestrin. Learning to (Learn at Test Time): RNNs with Expressive Hidden States, Aug. 2024. URL http: //arxiv.org/abs/2407.04620. arXiv:2407.04620. · arXiv
  95. I. Sutskever, O. Vinyals, and Q. V. Le. Sequence to Sequence Learning with Neural Networks. In Advances in Neural Information Processing Systems, volume 27.
  96. Curran Associates, Inc., 2014. URL https://proceedings.neurips. cc/paper/2014/hash/a14ac55a4f27472c5d894ec1c3c743d2-Abstract.html.
  97. K. Tanabe. Projection method for solving a singular system of linear equations and its applications. Numerische Mathematik, 17(3):203-214, June 1971. ISSN 0945-3245. doi: 10.1007/BF01436376. URL https://doi.org/10. 1007/BF01436376. · DOI
  98. Y. Tay, M. Dehghani, D. Bahri, and D. Metzler. Efficient Transformers: A Survey. ACM Comput. Surv., 55(6):109:1- 109:28, Dec. 2022. ISSN 0360-0300. doi: 10.1145/3530811. URL https://dl.acm.org/doi/10.1145/3530811. · DOI · Paper page
  99. A. Trockman, H. Harutyunyan, J. Z. Kolter, S. Kumar, and S. Bhojanapalli. Mimetic Initialization Helps State Space Models Learn to Recall, Oct. 2024. URL http://arxiv.org/abs/2410.11135. arXiv:2410.11135 [cs. · arXiv
  100. J. van der Westhuizen and J. Lasenby. The unreasonable effectiveness of the forget gate, Sept. 2018. URL http: //arxiv.org/abs/1804.04849. arXiv:1804.04849 [cs, stat]. · arXiv
  101. A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polo- sukhin. Attention is All you Need. In Advances in Neural Information Processing Systems, vol- ume 30. Curran Associates, Inc., 2017. URL https://papers.nips.cc/paper_files/paper/2017/hash/ 3f5ee243547dee91fbd053c1c4a845aa-Abstract.html.
  102. J. von Oswald, E. Niklasson, M. Schlegel, S. Kobayashi, N. Zucchet, N. Scherrer, N. Miller, M. Sandler, B. A. y Ar- cas, M. Vladymyrov, R. Pascanu, and J. Sacramento. Uncovering mesa-optimization algorithms in Transformers. CoRR, abs/2309.05858, 2023. URL https://doi.org/10.48550/arXiv.2309.05858. tex.cdate: 1672531200000 tex.publtype: informal. · DOI · Paper page
  103. K. A. Wang, G. Pleiss, J. Gardner, S. Tyree, K. Q. Weinberger, and A. G. Wilson. Exact Gaussian Processes on a Million Data Points. Advances in Neural Information Processing Systems, 32, 2019. URL https://papers.nips. cc/paper/2019/hash/01ce84968c6969bdd5d51c5eeaa3946a-Abstract.html.
  104. L. Wasserman. All of Nonparametric Statistics. Springer Texts in Statistics. Springer, New York, NY, 2006. ISBN 978-0-387-25145-5. doi: 10.1007/0-387-30623-4. URL http://link.springer.com/10.1007/0-387-30623-4. · DOI
  105. B. Widrow and M. E. Hoff. Adaptive switching circuits. In Neurocomputing: foundations of research, pages 123-134.
  106. D. Willshaw. Holography, Associative Memory, and Inductive Generalization. In Parallel Models of Associative Memory. Psychology Press, 1989. ISBN 978-1-315-80799-7. Num Pages: 22.
  107. D. J. Willshaw, O. P. Buneman, and H. C. Longuet-Higgins. Non-Holographic Associative Memory. Nature, 222 (5197):960-962, June 1969. ISSN 1476-4687. doi: 10.1038/222960a0. URL https://www.nature.com/articles/ 222960a0. Publisher: Nature Publishing Group. · DOI · Paper page
  108. M. Wortsman, P. J. Liu, L. Xiao, K. E. Everett, A. A. Alemi, B. Adlam, J. D. Co-Reyes, I. Gur, A. Kumar, R. Novak, J. Pennington, J. Sohl-Dickstein, K. Xu, J. Lee, J. Gilmer, and S. Kornblith. Small-scale proxies for large-scale Transformer training instabilities. In The Twelfth International Conference on Learning Representations, Oct. 2023. URL https://openreview.net/forum?id=d8w0pmvXbZ.
  109. M. Xu, W. Cheng, B. Wang, and W. Chen. KV Shifting Attention Enhances Language Modeling, Dec. 2024. URL http://arxiv.org/abs/2411.19574. arXiv:2411.19574 [cs. · arXiv
  110. S. Yang, J. Kautz, and A. Hatamizadeh. Gated Delta Networks: Improving Mamba2 with Delta Rule, Dec. 2024a. URL http://arxiv.org/abs/2412.06464. arXiv:2412.06464 [cs] version: 1. · arXiv
  111. S. Yang, B. Wang, Y. Shen, R. Panda, and Y. Kim. Gated Linear Attention Transformers with Hardware-Efficient Training. In Proceedings of the 41st International Conference on Machine Learning, pages 56501-56523. PMLR, July 2024b. URL https://proceedings.mlr.press/v235/yang24ab.html. ISSN: 2640-3498.
  112. S. Yang, B. Wang, Y. Zhang, Y. Shen, and Y. Kim. Parallelizing Linear Transformers with the Delta Rule over Sequence Length, Aug. 2024c. URL http://arxiv.org/abs/2406.06484. arXiv:2406.06484. · arXiv
  113. M. Zhang, K. Bhatia, H. Kumbong, and C. Re. The Hedgehog & the Porcupine: Expressive Linear Attentions with Softmax Mimicry. In The Twelfth International Conference on Learning Representations, Oct. 2023a. URL https: //openreview.net/forum?id=4g02l2N2Nx.
  114. Y. Zhang, F. Zhang, Z. Yang, and Z. Wang. What and How does In-Context Learning Learn? Bayesian Model Averaging, Parameterization, and Generalization, Oct. 2023b. URL http://arxiv.org/abs/2305.19420. arXiv:2305.19420 [stat. · arXiv
  115. Y. Zhang, B. Liu, Q. Cai, L. Wang, and Z. Wang. An Analysis of Attention via the Lens of Exchangeability and Latent Variable Models, Apr. 2024. URL http://arxiv.org/abs/2212.14852. arXiv:2212.14852 [cs. · arXiv
  116. J. Zhou and O. G. Troyanskaya. Predicting effects of noncoding variants with deep learning-based sequence model. Nature Methods, 12(10):931-934, Oct. 2015. ISSN 1548-7105. doi: 10.1038/nmeth.3547. URL https://www.nature. com/articles/nmeth.3547. Publisher: Nature Publishing Group. · DOI · Paper page
  117. M. Zinkevich. Online convex programming and generalized infinitesimal gradient ascent. In Proceedings of the Twen- tieth International Conference on International Conference on Machine Learning, ICML'03, pages 928-935, Wash- ington, DC, USA, Aug. 2003. AAAI Press. ISBN 978-1-57735-189-4.

Cited by

Sorted by citations

Authors
Wang, K. A., Shi, J., & Fox, E. B.
Affiliations
Stanford University (US)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2025
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2501.12352
arXiv
2501.12352
OpenAlex
W4406755454
Cited by
0
References
5
Type
posted content
Language
en
Open access
Licence
Locations
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Funders
  • National Science Foundation (2205084)
  • Office of Naval Research (N00014-22, FA9550, N00014)
  • Air Force Office of Scientific Research (FA9550-, -21-1-)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
Keywords
associative recall, sequence models, linear attention, softmax attention, state-space model, online students, transformer models, recurrent neural networks, unified framework

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review