CoChat
Sign Up
PreprintCited by 76Network Security and Intrusion DetectionNetwork Packet Processing and OptimizationInternet Traffic Analysis and Secure E-voting

Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality

Open access
Dao, Tri, Gu, Albert·2024·arXiv
DOI 10.48550/arxiv.2405.21060arXiv 2405.21060
Open this paper in CoChat — free accountOpen PDF

Abstract

While Transformers have been the main architecture behind deep learning's success in language modeling, state-space models (SSMs) such as Mamba have recently been shown to match or outperform Transformers at small to medium scale. We show that these families of models are actually quite closely related, and develop a rich framework of theoretical connections between SSMs and variants of attention, connected through various decompositions of a well-studied class of structured semiseparable matrices. Our state space duality (SSD) framework allows us to design a new architecture (Mamba-2) whose core layer is an a refinement of Mamba's selective SSM that is 2-8X faster, while continuing to be competitive with Transformers on language modeling.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2405.21060

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Recent recurrent and test-time training architectures use fixed-size states or fast-weight updates as sub-quadratic alternatives to full attention.

    “Recent recurrent and test-time training architectures use fixed-size states or fast-weight updates as sub-quadratic alternatives to full attention [21, 16, 35, 33, 20, 12, 52, 50].”

    From Learning What to Remember: Test-Time Training via Context Distillation (2026) · cites this paper as [12]

    Section 2.2 describes recent recurrent linear-attention variants and contrasts them with the quadratic cost of standard attention. Section 10.3 says SSD compresses sequence history into a controllable state rather than caching the full history, supporting the fixed-size-state, sub-quadratic part of the claim. The passages do not substantiate the additional characterization of test-time training architectures or fast-weight updates as alternatives to full attention. (LLM, conf=0.83)

    Experimental — accuracy not yet measured

  • Mamba 2 and Gated DeltaNet must use a linear hidden state and write custom kernels for efficient memory I/O.

    “while other prior work, such as Mamba 2 [21] and Gated DeltaNet [104], must use a linear hidden state and write custom kernels for efficient memory I/O.”

    From End-to-End Test-Time Training for Long Context (2025) · cites this paper as [21]

    Section 6 (page unavailable) describes the Mamba-2 SSD algorithm as combining a linear recurrent mode with an attention mode. But it explicitly says the algorithm is relatively efficient in native PyTorch without special low-level kernels, contrary to the claim that custom kernels are required. The [21] citation identifies Mamba-2; this paper does not establish the claim about Gated DeltaNet, which has a separate citation. (LLM, conf=0.85)

    Experimental — accuracy not yet measured

  • Using 5× the peak learning rate in GPT-3 and half the batch size for the 1B model improves the full attention baseline.

    “These two changes have been shown to improve the full attention baseline.”

    From End-to-End Test-Time Training for Long Context (2025) · cites this paper as [21]

    The Additional information section (page unavailable) says the authors used a peak learning rate 5× the GPT-3 value and reduced the 1.3B model’s batch size from 1M to 0.5M tokens. It calls the broader set of changes an “improved recipe,” but says the batch-size change was for uniformity and does not show that these two changes improved the full-attention baseline. (LLM, conf=0.86)

    Experimental — accuracy not yet measured

  • Prior work must fit their hidden states onto the individual chips inside a GPU, which significantly limits their hidden state size.

    “In contrast, prior work must fit their hidden states onto the individual chips inside a GPU, which significantly limits their hidden state size.”

    From End-to-End Test-Time Training for Long Context (2025) · cites this paper as [21]

    Figure 10 says that increasing state expansion slows Mamba’s optimized scan, while SSD handles much larger expansion factors without much slowdown. The paper thus supports a practical constraint on prior work’s state size, but the passages do not say that hidden states must fit on individual GPU chips or identify that as the cause. (LLM, conf=0.83)

    Experimental — accuracy not yet measured

  • The SSD framework is limited to models with vector or matrix-valued memory updated using Hebbian-like update rules.

    “The SSD framework, however, is limited to models with vector or matrix-valued memory that are updated using a Hebbian-like update rules.”

    From It's All Connected: A Journey Through Test-Time Memorization, Attentional Bias, Retention, and Online Optimization (2025) · cites this paper as Dao et al. (2024)

    Section 2.4 describes SSD as a recurrent state-space model, and the hardware-efficient algorithm section describes its expanded hidden state as an (N, P) tensor. But the paper identifies SSD’s main restriction as scalar-identity state transitions (Structure; §5.3), not a requirement for Hebbian-like update rules. The claimed limitation on update rules is therefore unsupported. (LLM, conf=0.82)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

No references could be resolved yet.

Research works printed but not linked: 112

  1. Joshua Ainslie, James Lee-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebrón, and Sumit Sanghai. "GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints". In: arXiv preprint arXiv:2305.13245 (2023). · arXiv
  2. Yaroslav Aksenov, Nikita Balagansky, Sofia Maria Lo Cicero Vaina, Boris Shaposhnikov, Alexey Gorbatovski, and Daniil Gavrilov. "Linear Transformers with Learnable Kernel Functions are Better In-Context Models". In: arXiv preprint arXiv:2402.10644 (2024). · arXiv
  3. Ekin Akyürek, Bailin Wang, Yoon Kim, and Jacob Andreas. "In-Context Language Learning: Architectures and Algorithms". In: The International Conference on Machine Learning (ICML). 2024.
  4. Ameen Ali, Itamar Zimerman, and Lior Wolf. The Hidden Attention of Mamba Models. 2024. arXiv: 2403.01590 [cs.LG]. · arXiv
  5. Simran Arora, Sabri Eyuboglu, Aman Timalsina, Isys Johnson, Michael Poli, James Zou, Atri Rudra, and Christo- pher Ré. "Zoology: Measuring and Improving Recall in Efficient Language Models". In: The International Conference on Learning Representations (ICLR). 2024.
  6. Simran Arora, Sabri Eyuboglu, Michael Zhang, Aman Timalsina, Silas Alberti, Dylan Zinsley, James Zou, Atri Rudra, and Christopher Ré. "Simple Linear Attention Language Models Balance the Recall-Throughput Tradeoff". In: The International Conference on Machine Learning (ICML). 2024.
  7. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. "Neural Machine Translation by Jointly Learning to Align and Translate". In: The International Conference on Learning Representations (ICLR). 2015.
  8. George A Baker, George A Baker Jr, Peter Graves-Morris, and Susan S Baker. Pade Approximants: Encyclopedia of Mathematics and It's Applications, Vol. 59
  9. George A. Baker, Jr., Peter Graves-Morris. Vol. 59. Cambridge University Press, 1996.
  10. Maximilian Beck, Korbinian Pöppel, Markus Spanring, Andreas Auer, Oleksandra Prudnikova, Michael Kopp, Gün- ter Klambauer, Johannes Brandstetter, and Sepp Hochreiter. "xLSTM: Extended Long Short-Term Memory". In: arXiv preprint arXiv:2405.04517 (2024). · arXiv
  11. Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O'Brien, Eric Hallahan, Mo- hammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al. "Pythia: A Suite for Analyzing Large Language Models across Training and Scaling". In: The International Conference on Machine Learning (ICML). PMLR. 2023, pp. 2397-2430.
  12. Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. "PIQA: Reasoning about Physical Commonsense in Natural Language". In: Proceedings of the AAAI conference on Artificial Intelligence. Vol. 34. 2020.
  13. Sid Black, Stella Biderman, Eric Hallahan, Quentin Anthony, Leo Gao, Laurence Golding, Horace He, Connor Leahy, Kyle McDonell, Jason Phang, et al. "Gpt-NeoX-20B: An Open-source Autoregressive Language Model". In: arXiv preprint arXiv:2204.06745 (2022). · arXiv
  14. Aleksandar Botev, Soham De, Samuel L Smith, Anushan Fernando, George-Cristian Muraru, Ruba Haroun, Leonard Berrada, Razvan Pascanu, Pier Giuseppe Sessa, Robert Dadashi, et al. "RecurrentGemma: Moving Past Transform- ers for Efficient Open Language Models". In: arXiv preprint arXiv:2404.07839 (2024). · arXiv
  15. George EP Box, Gwilym M Jenkins, Gregory C Reinsel, and Greta M Ljung. Time Series Analysis: Forecasting and Control. John Wiley & Sons, 2015.
  16. James Bradbury, Stephen Merity, Caiming Xiong, and Richard Socher. "Quasi-recurrent Neural Networks". In: arXiv preprint arXiv:1611.01576 (2016). · arXiv
  17. William Brandon, Aniruddha Nrusimha, Kevin Qian, Zachary Ankner, Tian Jin, Zhiye Song, and Jonathan Ragan- Kelley. "Striped attention: Faster ring attention for causal transformers". In: arXiv preprint arXiv:2311.09431 (2023). · arXiv
  18. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakan- tan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. "Language Models are Few-shot Learners". In: Advances in Neural Information Processing Systems (NeurIPS) 33 (2020), pp. 1877-1901.
  19. Krzysztof Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Davis, Afroz Mohiuddin, Lukasz Kaiser, et al. "Rethinking Attention with Performers". In: The International Conference on Learning Representations (ICLR). 2021.
  20. Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. "PaLM: Scaling Language Modeling with Path- ways". In: Journal of Machine Learning Research 24.240 (2023), pp. 1-113. url: http://jmlr.org/papers/v24/22- 1144.html.
  21. Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. "Empirical Evaluation of Gated Recur- rent Neural Networks on Sequence Modeling". In: arXiv preprint arXiv:1412.3555 (2014). · arXiv
  22. Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. "Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge". In: arXiv preprint arXiv:1803.05457 (2018). · arXiv
  23. Tri Dao. "FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning". In: The International Conference on Learning Representations (ICLR). 2024.
  24. Tri Dao, Beidi Chen, Nimit S Sohoni, Arjun Desai, Michael Poli, Jessica Grogan, Alexander Liu, Aniruddh Rao, Atri Rudra, and Christopher Ré. "Monarch: Expressive structured matrices for efficient and accurate training". In: International Conference on Machine Learning. PMLR. 2022, pp. 4690-4721.
  25. Tri Dao, Daniel Y Fu, Khaled K Saab, Armin W Thomas, Atri Rudra, and Christopher Ré. "Hungry Hungry Hippos: Towards Language Modeling with State Space Models". In: The International Conference on Learning Representations (ICLR). 2023.
  26. Tri Dao, Albert Gu, Matthew Eichhorn, Atri Rudra, and Christopher Ré. "Learning Fast Algorithms for Linear Transforms Using Butterfly Factorizations". In: The International Conference on Machine Learning (ICML). 2019.
  27. Tri Dao, Nimit Sohoni, Albert Gu, Matthew Eichhorn, Amit Blonder, Megan Leszczynski, Atri Rudra, and Christo- pher Ré. "Kaleidoscope: An Efficient, Learnable Representation for All Structured Linear Maps". In: The Interna- tional Conference on Learning Representations (ICLR). 2020.
  28. Timothée Darcet, Maxime Oquab, Julien Mairal, and Piotr Bojanowski. "Vision Transformers Need Registers". In: The International Conference on Learning Representations (ICLR). 2024.
  29. Soham De, Samuel L Smith, Anushan Fernando, Aleksandar Botev, George Cristian-Muraru, Albert Gu, Ruba Haroun, Leonard Berrada, Yutian Chen, Srivatsan Srinivasan, et al. "Griffin: Mixing Gated Linear Recurrences with Local Attention for Efficient Language Models". In: arXiv preprint arXiv:2402.19427 (2024). · arXiv
  30. Christopher De Sa, Albert Gu, Rohan Puttagunta, Christopher Ré, and Atri Rudra. "A Two-Pronged Progress in Structured Dense Matrix Vector Multiplication". In: Proceedings of the Twenty-Ninth Annual ACM-SIAM Symposium on Discrete Algorithms. SIAM. 2018, pp. 1060-1079.
  31. Hantian Ding, Zijian Wang, Giovanni Paolini, Varun Kumar, Anoop Deoras, Dan Roth, and Stefano Soatto. "Fewer truncations improve language modeling". In: arXiv preprint arXiv:2404.10830 (2024). · arXiv
  32. Yuli Eidelman and Israel Gohberg. "On a new class of structured matrices". In: Integral Equations and Operator Theory 34.3 (1999), pp. 293-324.
  33. Dan Fu, Simran Arora, Jessica Grogan, Isys Johnson, Evan Sabri Eyuboglu, Armin Thomas, Benjamin Spector, Michael Poli, Atri Rudra, and Christopher Ré. "Monarch mixer: A simple sub-quadratic gemm-based architecture". In: Advances in Neural Information Processing Systems 36 (2024).
  34. Daniel Y Fu, Elliot L Epstein, Eric Nguyen, Armin W Thomas, Michael Zhang, Tri Dao, Atri Rudra, and Christopher Ré. "Simple Hardware-efficient Long Convolutions for Sequence Modeling". In: The International Conference on Machine Learning (ICML) (2023).
  35. Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. "The Pile: An 800GB Dataset of Diverse Text for Language Modeling". In: arXiv preprint arXiv:2101.00027 (2020). · arXiv
  36. Paolo Glorioso, Quentin Anthony, Yury Tokpanov, James Whittington, Jonathan Pilault, Adam Ibrahim, and Beren Millidge. "Zamba: A Compact 7B SSM Hybrid Model". In: arXiv preprint arXiv:2405.16712 (2024). · arXiv
  37. Riccardo Grazzi, Julien Siems, Simon Schrodi, Thomas Brox, and Frank Hutter. "Is Mamba Capable of In-Context Learning?" In: arXiv preprint arXiv:2402.03170 (2024). · arXiv
  38. Albert Gu. "Modeling Sequences with Structured State Spaces". PhD thesis. Stanford University, 2023.
  39. Albert Gu and Tri Dao. "Mamba: Linear-Time Sequence Modeling with Selective State Spaces". In: arXiv preprint arXiv:2312.00752 (2023). · arXiv
  40. Albert Gu, Tri Dao, Stefano Ermon, Atri Rudra, and Christopher Ré. "HIPPO: Recurrent Memory with Optimal Polynomial Projections". In: Advances in Neural Information Processing Systems (NeurIPS). 2020.
  41. Albert Gu, Karan Goel, and Christopher Ré. "Efficiently Modeling Long Sequences with Structured State Spaces". In: The International Conference on Learning Representations (ICLR). 2022.
  42. Albert Gu, Ankit Gupta, Karan Goel, and Christopher Ré. "On the Parameterization and Initialization of Diagonal State Space Models". In: Advances in Neural Information Processing Systems (NeurIPS). 2022.
  43. Albert Gu, Isys Johnson, Karan Goel, Khaled Saab, Tri Dao, Atri Rudra, and Christopher Ré. "Combining Recurrent, Convolutional, and Continuous-time Models with the Linear State Space Layer". In: Advances in Neural Information Processing Systems (NeurIPS). 2021.
  44. Albert Gu, Isys Johnson, Aman Timalsina, Atri Rudra, and Christopher Ré. "How to Train Your HIPPO: State Space Models with Generalized Basis Projections". In: The International Conference on Learning Representations (ICLR). 2023.
  45. Ankit Gupta, Albert Gu, and Jonathan Berant. "Diagonal State Spaces are as Effective as Structured State Spaces". In: Advances in Neural Information Processing Systems 35 (2022), pp. 22982-22994.
  46. Dan Hendrycks and Kevin Gimpel. "Gaussian Error Linear Units (GELUs)". In: arXiv preprint arXiv:1606.08415 (2016). · arXiv
  47. W Daniel Hillis and Guy L Steele Jr. "Data Parallel Algorithms". In: Communications of the ACM 29.12 (1986), pp. 1170-1183.
  48. Sepp Hochreiter and Jürgen Schmidhuber. "Long Short-Term Memory". In: Neural Computation 9.8 (1997), pp. 1735- 1780.
  49. Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. "An Empirical Analysis of Compute- Optimal Large Language Model Training". In: Advances in Neural Information Processing Systems (NeurIPS) 35 (2022), pp. 30016-30030.
  50. Samy Jelassi, David Brandfonbrener, Sham M Kakade, and Eran Malach. "Repeat After Me: Transformers Are Better Than State Space Models at Copying". In: The International Conference on Machine Learning (ICML). 2024.
  51. Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret. "Transformers are RNNs: Fast Au- toregressive Transformers with Linear Attention". In: International Conference on Machine Learning. PMLR. 2020, pp. 5156-5165.
  52. Tobias Katsch. "GateLoop: Fully Data-Controlled Linear Recurrence for Sequence Modeling". In: arXiv preprint arXiv:2311.01927 (2023). · arXiv
  53. Shiva Kaul. "Linear Dynamical Systems as a Core Computational Primitive". In: Advances in Neural Information Processing Systems 33 (2020), pp. 16808-16820.
  54. Vijay Anand Korthikanti, Jared Casper, Sangkug Lym, Lawrence McAfee, Michael Andersch, Mohammad Shoeybi, and Bryan Catanzaro. "Reducing activation recomputation in large transformer models". In: Proceedings of Machine Learning and Systems 5 (2023).
  55. James Lee-Thorp, Joshua Ainslie, Ilya Eckstein, and Santiago Ontanon. "Fnet: Mixing tokens with fourier trans- forms". In: arXiv preprint arXiv:2105.03824 (2021). · arXiv
  56. Tao Lei. "When Attention Meets Fast Recurrence: Training Language Models with Reduced Compute". In: Pro- ceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. 2021, pp. 7633-7648.
  57. Tao Lei, Yu Zhang, Sida I Wang, Hui Dai, and Yoav Artzi. "Simple Recurrent Units for Highly Parallelizable Recur- rence". In: arXiv preprint arXiv:1709.02755 (2017). · arXiv
  58. Yuhong Li, Tianle Cai, Yi Zhang, Deming Chen, and Debadeepta Dey. "What Makes Convolutional Models Great on Long Sequence Modeling?" In: The International Conference on Learning Representations (ICLR). 2023.
  59. Opher Lieber, Barak Lenz, Hofit Bata, Gal Cohen, Jhonathan Osin, Itay Dalmedigos, Erez Safahi, Shaked Meirom, Yonatan Belinkov, Shai Shalev-Shwartz, et al. "Jamba: A Hybrid Transformer-Mamba Language Model". In: arXiv preprint arXiv:2403.19887 (2024). · arXiv
  60. Hao Liu, Wilson Yan, Matei Zaharia, and Pieter Abbeel. "World Model on Million-Length Video And Language With RingAttention". In: arXiv preprint arXiv:2402.08268 (2024). · arXiv
  61. Hao Liu, Matei Zaharia, and Pieter Abbeel. "Ring attention with blockwise transformers for near-infinite context". In: arXiv preprint arXiv:2310.01889 (2023). · arXiv
  62. Chris Lu, Yannick Schroecker, Albert Gu, Emilio Parisotto, Jakob Foerster, Satinder Singh, and Feryal Behbahani. "Structured State Space Models for In-Context Reinforcement Learning". In: Advances in Neural Information Pro- cessing Systems (NeurIPS). 2023.
  63. Xuezhe Ma, Chunting Zhou, Xiang Kong, Junxian He, Liangke Gui, Graham Neubig, Jonathan May, and Luke Zettlemoyer. "Mega: Moving Average Equipped Gated Attention". In: The International Conference on Learning Representations (ICLR). 2023.
  64. Eric Martin and Chris Cundy. "Parallelizing Linear Recurrent Neural Nets Over Sequence Length". In: The Inter- national Conference on Learning Representations (ICLR). 2018.
  65. Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. "Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering". In: arXiv preprint arXiv:1809.02789 (2018). · arXiv
  66. Antonio Orvieto, Samuel L Smith, Albert Gu, Anushan Fernando, Caglar Gulcehre, Razvan Pascanu, and Soham De. "Resurrecting Recurrent Neural Networks for Long Sequences". In: The International Conference on Machine Learning (ICML). 2023.
  67. Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Ngoc-Quan Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernández. "The LAMBADA Dataset: Word Prediction Requiring a Broad Discourse Context". In: Proceedings of the 54th Annual Meeting of the Association for Computational Linguis- tics. 2016, pp. 1525-1534.
  68. Jongho Park, Jaeseung Park, Zheyang Xiong, Nayoung Lee, Jaewoong Cho, Samet Oymak, Kangwook Lee, and Dimitris Papailiopoulos. "Can Mamba Learn How to Learn? A Comparative Study on In-Context Learning Tasks". In: The International Conference on Machine Learning (ICML). 2024.
  69. Bo Peng, Eric Alcaide, Quentin Anthony, Alon Albalak, Samuel Arcadinho, Huanqi Cao, Xin Cheng, Michael Chung, Matteo Grella, Kranthi Kiran GV, et al. "RWKV: Reinventing RNNs for the Transformer Era". In: arXiv preprint arXiv:2305.13048 (2023). · arXiv
  70. Bo Peng, Daniel Goldstein, Quentin Anthony, Alon Albalak, Eric Alcaide, Stella Biderman, Eugene Cheah, Teddy Ferdinan, Haowen Hou, Przemysław Kazienko, et al. "Eagle and Finch: RWKV with matrix-valued states and dy- namic recurrence". In: arXiv preprint arXiv:2404.05892 (2024). · arXiv
  71. Hao Peng, Nikolaos Pappas, Dani Yogatama, Roy Schwartz, Noah A Smith, and Lingpeng Kong. "Random Feature Attention". In: The International Conference on Learning Representations (ICLR). 2021.
  72. Clément Pernet. "Computing with Quasiseparable Matrices". In: Proceedings of the ACM on International Sympo- sium on Symbolic and Algebraic Computation. 2016, pp. 389-396.
  73. Clément Pernet, Hippolyte Signargout, and Gilles Villard. "Exact computations with quasiseparable matrices". In: arXiv preprint arXiv:2302.04515 (2023). · arXiv
  74. Clément Pernet and Arne Storjohann. "Time and space efficient generators for quasiseparable matrices". In: Journal of Symbolic Computation 85 (2018), pp. 224-246.
  75. Michael Poli, Stefano Massaroli, Eric Nguyen, Daniel Y Fu, Tri Dao, Stephen Baccus, Yoshua Bengio, Stefano Ermon, and Christopher Ré. "Hyena Hierarchy: Towards Larger Convolutional Language Models". In: The International Conference on Machine Learning (ICML). 2023.
  76. Hadi Pouransari, Chun-Liang Li, Jen-Hao Rick Chang, Pavan Kumar Anasosalu Vasu, Cem Koc, Vaishaal Shankar, and Oncel Tuzel. "Dataset Decomposition: Faster LLM Training with Variable Sequence Length Curriculum". In: arXiv preprint arXiv:2405.13226 (2024). · arXiv
  77. Ofir Press, Noah Smith, and Mike Lewis. "Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation". In: International Conference on Learning Representations. 2022.
  78. Zhen Qin, Xiaodong Han, Weixuan Sun, Bowen He, Dong Li, Dongxu Li, Yuchao Dai, Lingpeng Kong, and Yiran Zhong. "Toeplitz Neural Network for Sequence Modeling". In: The International Conference on Learning Represen- tations (ICLR). 2023.
  79. Zhen Qin, Xiaodong Han, Weixuan Sun, Dongxu Li, Lingpeng Kong, Nick Barnes, and Yiran Zhong. "The devil in linear transformer". In: arXiv preprint arXiv:2210.10340 (2022). · arXiv
  80. Zhen Qin, Dong Li, Weigao Sun, Weixuan Sun, Xuyang Shen, Xiaodong Han, Yunshen Wei, Baohong Lv, Xiao Luo, Yu Qiao, et al. "TransNormerLLM: A Faster and Better Large Language Model with Improved TransNormer". In: arXiv preprint arXiv:2307.14995 (2023). · arXiv
  81. Zhen Qin, Weixuan Sun, Hui Deng, Dongxu Li, Yunshen Wei, Baohong Lv, Junjie Yan, Lingpeng Kong, and Yiran Zhong. "CosFormer: Rethinking Softmax in Attention". In: The International Conference on Learning Representations (ICLR). 2022.
  82. Zhen Qin, Songlin Yang, Weixuan Sun, Xuyang Shen, Dong Li, Weigao Sun, and Yiran Zhong. "HGRN2: Gated Linear RNNs with State Expansion". In: arXiv preprint arXiv:2404.07904 (2024). · arXiv
  83. Zhen Qin, Songlin Yang, and Yiran Zhong. "Hierarchically Gated Recurrent Neural Network for Sequence Model- ing". In: Advances in Neural Information Processing Systems 36 (2023).
  84. Ali Rahimi and Benjamin Recht. "Random Features for Large-Scale Kernel Machines". In: Advances in Neural In- formation Processing Systems (NeurIPS) 20 (2007).
  85. Prajit Ramachandran, Barret Zoph, and Quoc V Le. "Swish: A Self-gated Activation Function". In: arXiv preprint arXiv:1710.05941 7.1 (2017), p. 5. · arXiv
  86. Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. "Winogrande: An Adversarial Winograd Schema Challenge at Scale". In: Communications of the ACM 64.9 (2021), pp. 99-106.
  87. Imanol Schlag, Kazuki Irie, and Jürgen Schmidhuber. "Linear Transformers are Secretly Fast Weight Programmers". In: The International Conference on Machine Learning (ICML). PMLR. 2021, pp. 9355-9366.
  88. Noam Shazeer. "Fast Transformer Decoding: One Write-head is All You Need". In: arXiv preprint arXiv:1911.02150 (2019). · arXiv
  89. Sam Shleifer, Jason Weston, and Myle Ott. "NormFormer: Improved Transformer Pretraining with Extra Normal- ization". In: arXiv preprint arXiv:2110.09456 (2021). · arXiv
  90. Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. "Megatron- LM: Training Multi-Billion Parameter Language Models Using Model Parallelism". In: arXiv preprint arXiv:1909.08053 (2019). · arXiv
  91. Jimmy TH Smith, Andrew Warrington, and Scott W Linderman. "Simplified State Space Layers for Sequence Mod- eling". In: The International Conference on Learning Representations (ICLR). 2023.
  92. Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu. "Roformer: Enhanced Transformer with Rotary Position Embedding". In: arXiv preprint arXiv:2104.09864 (2021). · arXiv
  93. Yutao Sun, Li Dong, Shaohan Huang, Shuming Ma, Yuqing Xia, Jilong Xue, Jianyong Wang, and Furu Wei. "Reten- tive network: A successor to transformer for large language models". In: arXiv preprint arXiv:2307.08621 (2023). · arXiv
  94. Yi Tay, Mostafa Dehghani, Dara Bahri, and Donald Metzler. "Efficient Transformers: A Survey". In: ACM Comput- ing Surveys 55.6 (2022), pp. 1-28.
  95. Chameleon Team. "Chameleon: Mixed-Modal Early-Fusion Foundation Models". In: arXiv preprint arXiv:2405.09818 (2024). · arXiv
  96. Anna Thomas, Albert Gu, Tri Dao, Atri Rudra, and Christopher Ré. "Learning Compressed Transforms with Low Displacement Rank". In: Advances in Neural Information Processing Systems (NeurIPS). 2018, pp. 9052-9060.
  97. Ilya O Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Thomas Unterthiner, Jessica Yung, Andreas Steiner, Daniel Keysers, Jakob Uszkoreit, et al. "MLP-Mixer: An All-MLP Architecture for Vision". In: Advances in Neural Information Processing Systems (NeurIPS) 34 (2021), pp. 24261-24272.
  98. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. "Llama: Open and Efficient Foundation Language Models". In: arXiv preprint arXiv:2302.13971 (2023). · arXiv
  99. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. "Llama 2: Open foundation and fine-tuned chat models". In: arXiv preprint arXiv:2307.09288 (2023). · arXiv
  100. Raf Vandebril, M Van Barel, Gene Golub, and Nicola Mastronardi. "A bibliography on semiseparable matrices". In: Calcolo 42 (2005), pp. 249-270.
  101. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. "Attention Is All You Need". In: Advances in Neural Information Processing Systems (NeurIPS). 2017.
  102. Shida Wang and Beichen Xue. "State-space Models with Layer-wise Nonlinearity are Universal Approximators with Exponential Decaying Memory". In: arXiv preprint arXiv:2309.13414 (2023). · arXiv
  103. Sinong Wang, Belinda Z Li, Madian Khabsa, Han Fang, and Hao Ma. "Linformer: Self-attention with Linear Com- plexity". In: arXiv preprint arXiv:2006.04768 (2020). · arXiv
  104. Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. "Efficient Streaming Language Models with Attention Sinks". In: The International Conference on Learning Representations (ICLR). 2024.
  105. Yunyang Xiong, Zhanpeng Zeng, Rudrasis Chakraborty, Mingxing Tan, Glenn Fung, Yin Li, and Vikas Singh. "Nyströmformer: A Nyström-Based Algorithm for Approximating Self-Attention". In: Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 35. 2021.
  106. Songlin Yang, Bailin Wang, Yikang Shen, Rameswar Panda, and Yoon Kim. "Gated Linear Attention Transformers with Hardware-Efficient Training". In: The International Conference on Machine Learning (ICML). 2024.
  107. Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. "HellaSwag: Can a Machine Really Finish Your Sentence?" In: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. 2019.
  108. Jinle Zeng, Min Li, Zhihua Wu, Jiaqi Liu, Yuang Liu, Dianhai Yu, and Yanjun Ma. "Boosting distributed training performance of the unpadded bert model". In: arXiv preprint arXiv:2208.08124 (2022). · arXiv
  109. Shuangfei Zhai, Walter Talbott, Nitish Srivastava, Chen Huang, Hanlin Goh, Ruixiang Zhang, and Josh Susskind. "An Attention Free Transformer". In: arXiv preprint arXiv:2105.14103 (2021). · arXiv
  110. Yujia Zhai, Chengquan Jiang, Leyuan Wang, Xiaoying Jia, Shang Zhang, Zizhong Chen, Xin Liu, and Yibo Zhu. "Bytetransformer: A high-performance transformer boosted for variable-length inputs". In: 2023 IEEE International Parallel and Distributed Processing Symposium (IPDPS). IEEE. 2023, pp. 344-355.
  111. Michael Zhang, Kush Bhatia, Hermann Kumbong, and Christopher Ré. "The Hedgehog & the Porcupine: Expres- sive Linear Attentions with Softmax Mimicry". In: The International Conference on Learning Representations (ICLR). 2024.
  112. Lin Zheng, Chong Wang, and Lingpeng Kong. "Linear complexity randomized self-attention mechanism". In: In- ternational Conference on Machine Learning. PMLR. 2022, pp. 27011-27041.

Cited by

Sorted by citations

Authors
Dao, T., & Gu, A.
Affiliations
  • Princeton University (US)
  • Carnegie Mellon University (US)
Venue
arXiv
Publisher
arXiv
Published
2024
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2405.21060
arXiv
2405.21060
OpenAlex
W4399317989
Cited by
76
References
0
Citations by year
2026: 16 · 2025: 50 · 2024: 10
Type
posted content
Language
en
Open access
Licence
Locations
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Topics
  • Physical Sciences › Computer Science › Computer Networks and Communications › Network Security and Intrusion Detection
  • Physical Sciences › Computer Science › Hardware and Architecture › Network Packet Processing and Optimization
  • Physical Sciences › Computer Science › Artificial Intelligence › Internet Traffic Analysis and Secure E-voting
Keywords
Mamba-2, selective state space models, Transformer comparison, extreme value theory, intrusion detection, false positive rate, threshold selection

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review