The DOI matches its DataCite record; 1 of the 1 references it cites were found in a registry or an index.
Simplified State Space Layers for Sequence Modeling
Open accessAbstract
Models using structured state space sequence (S4) layers have achieved state-of-the-art performance on long-range sequence modeling tasks. An S4 layer combines linear state space models (SSMs), the HiPPO framework, and deep learning to achieve high performance. We build on the design of the S4 layer and introduce a new state space layer, the S5 layer. Whereas an S4 layer uses many independent single-input, single-output SSMs, the S5 layer uses one multi-input, multi-output SSM. We establish a connection between S5 and S4, and use this to develop the initialization and parameterization used by the S5 model. The result is a state space layer that can leverage efficient and widely implemented parallel scans, allowing S5 to match the computational efficiency of S4, while also achieving state-of-the-art performance on several long-range sequence modeling tasks. S5 averages 87.4% on the long range arena benchmark, and 98.5% on the most difficult Path-X task.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Rethinking Attention with Performers
Choromanski, K., Likhosherstov, V., Dohan, D., Song, X., Gane, A., Sarlos, T., Hawkins, P., Davis, J., Mohiuddin, A., Kaiser, L., et al. (2020) · arXiv · doi:10.48550/arxiv.2009.14794 · Cited by 695 · cited 2×
“In particular, many efficient transformer methods have been introduced [11, 27, 29, 5, 21, 59] to address the standard transformer’s quadratic complexity in the sequence length.”
BackgroundMethod
Flagged first, then by citations
Cited by
Sorted by citations
Mamba: Linear-Time Sequence Modeling with Selective State Spaces
Gu, A., & Dao, T. (2023) · arXiv · doi:10.48550/arxiv.2312.00752 · Cited by 1,579
xLSTM: Extended Long Short-Term Memory
Beck, M., Pöppel, K., Spanring, M., Auer, A., Prudnikova, O., Kopp, M., Klambauer, G., Brandstetter, J., & Hochreiter, S. (2024) · Advances in Neural Information Processing Systems 37 · doi:10.52202/079017-3417 · Cited by 286
xLSTM: Extended Long Short-Term Memory
Beck, M., Pöppel, K., Spanring, M., Auer, A., Prudnikova, O., Kopp, M., Klambauer, G., Brandstetter, J., & Hochreiter, S. (2024) · arXiv · doi:10.48550/arxiv.2405.04517 · Cited by 299
VMRNN: Integrating Vision Mamba and LSTM for Efficient and Accurate Spatiotemporal Forecasting
Tang, Y., Dong, P., Tang, Z., Chu, X., & Liang, J. (2024) · 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw63382.2024.00575 · Cited by 70
A Survey: Collaborative Hardware and Software Design in the Era of Large Language Models
Guo, C., Cheng, F., Du, Z., Kiessling, J., Ku, J., Li, S., Li, Z., Ma, M., Molom-Ochir, T., Morris, B., et al. (2025) · IEEE Circuits and Systems Magazine · doi:10.1109/mcas.2024.3476008 · Cited by 32
Titans: Learning to Memorize at Test Time
Behrouz, A., Zhong, P., & Mirrokni, V. (2025) · arXiv · doi:10.48550/arxiv.2501.00663 · Cited by 24
“Accordingly, we can use parallel associative scan [37] to calculate S t s in each chunk.”
Nested Learning: The Illusion of Deep Learning Architectures
Behrouz, A., Razaviyayn, M., Zhong, P., & Mirrokni, V. (2025) · arXiv · doi:10.48550/arxiv.2512.24695 · Cited by 37
“Initial models, including RetNet [61], RWKV [77], and S5 [78], employed data-independent transition matrices with Hebbian-like update rules.”
Mamba-3: Improved Sequence Modeling using State Space Principles
Lahoti, A., Li, K. Y., Chen, B., Wang, C., Bick, A., Kolter, J. Z., Dao, T., & Gu, A. (2026) · arXiv · doi:10.48550/arxiv.2603.15569 · Cited by 1
“where L ∈ R 𝑇 × 𝑇 is a structured mask, B , C ∈ R 𝑇 × 𝑁 , X ∈ R 𝑇 × 𝐷 are the inputs to the SSM and Y ∈ R 𝑇 × 𝐷 Different structures on L give rise to various instantiations of SSD.”
Gated DeltaNet-2: Decoupling Erase and Write in Linear Attention
Hatamizadeh, A., Choi, Y., & Kautz, J. (2026) · arXiv
“Early structured state-space and recurrent models used mostly data-independent transitions [16, 38, 39, 40], while Mamba and Mamba-2 introduced data-dependent selective dynamics and the SSD framework [8, 41].”
Behrouz, A., Razaviyayn, M., Zhong, P., & Mirrokni, V. (2025) · arXiv · doi:10.48550/arxiv.2504.13173 · Cited by 65
Gopal, A. M., Pirbadian, A., Carlson, K. D., Lewis, M. A., & Tapson, J. (2026) · arXiv
“S4 [1], S5 [2], and Mamba [3, 4] established structured and selective SSMs as Transformer-competitive sequence models; RWKV [5] and RetNet [6] explore related linear-recurrent designs.”
Artificial Generational Intelligence: Cultural Accumulation in Reinforcement Learning
Cook, J., Lu, C., Hughes, E., Leibo, J. Z., & Foerster, J. (2024) · arXiv · doi:10.48550/arxiv.2406.00392 · Cited by 6
“For each of our experiments, we use a Simplified Structured State Space Model (S5) [Smith et al., 2022] modified for RL [Lu et al., 2024] to encode memory, building on the PureJaxRL codebase [Lu et al., 2022a].”
MethodATLAS: Learning to Optimally Memorize the Context at Test Time
Behrouz, A., Li, Z., Kacham, P., Daliri, M., Deng, Y., Zhong, P., Razaviyayn, M., & Mirrokni, V. (2025) · arXiv · doi:10.48550/arxiv.2505.23735
Implicit Language Models are RNNs: Balancing Parallelization and Expressivity
Schöne, M., Rahmani, B., Kremer, H., Falck, F., Ballani, H., & Gladrow, J. (2025) · arXiv · doi:10.48550/arxiv.2502.07827
“The learned matrix Λ ∈ R n × n is typically diagonal and can be constant (Gu et al., 2022; Smith et al., 2023) or an input-dependent matrix-valued function (Qin et al., 2023; Gu & Dao, 2023; Dao & Gu, 2024).”
Metadata
- Authors
- Smith, J. T. H., Warrington, A., & Linderman, S. W.
- Affiliations
- Stanford University (US)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2022
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2208.04933
- OpenAlex
- W4290802752
- Cited by
- 162
- References
- 1
- Citations by year
- 2026: 6 · 2025: 104 · 2024: 45 · 2023: 7
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University) (cc-by)
- Funders
- Alfred P. Sloan Foundation
- National Institutes of Health
- National Institute of Neurological Disorders and Stroke (R01NS113119, U19NS113201)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Signal Processing › Time Series Analysis and Forecasting
- Physical Sciences › Computer Science › Artificial Intelligence › Neural Networks and Applications
- Keywords
- structured state space models, parallel scan, parameter initialization
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
