CoChat

The DOI matches its Crossref record; 99 of the 101 references it cites were found in a registry or an index; 19 of the 35 sources its claims cite were checked for support.

xLSTM: Extended Long Short-Term Memory

Open access

Abstract

In the 1990s, the constant error carousel and gating were introduced as the central ideas of the Long Short-Term Memory (LSTM). Since then, LSTMs have stood the test of time and contributed to numerous deep learning success stories, in particular they constituted the first Large Language Models (LLMs). However, the advent of the Transformer technology with parallelizable self-attention at its core marked the dawn of a new era, outpacing LSTMs at scale. We now raise a simple question: How far do we get in language modeling when scaling LSTMs to billions of parameters, leveraging the latest techniques from modern LLMs, but mitigating known limitations of LSTMs? Firstly, we introduce exponential gating with appropriate normalization and stabilization techniques. Secondly, we modify the LSTM memory structure, obtaining: (i) sLSTM with a scalar memory, a scalar update, and new memory mixing, (ii) mLSTM that is fully parallelizable with a matrix memory and a covariance update rule. Integrating these LSTM extensions into residual block backbones yields xLSTM blocks that are then residually stacked into xLSTM architectures. Exponential gating and modified memory structures boost xLSTM capabilities to perform favorably when compared to state-of-the-art Transformers and State Space Models, both in performance and scaling.

Overview

CoChat Summary

TL;DR

xLSTM extends LSTM with exponential gating and new memory structures—sLSTM (scalar memory, memory mixing) and mLSTM (matrix memory, covariance update, parallelizable)—and shows favorable language-modeling performance versus Transformers and State Space Models. On 15B and 300B SlimPajama tokens, xLSTM variants achieve the best validation perplexity among compared methods and extrapolate better to longer contexts.

Key findings

  • On 15B SlimPajama tokens, xLSTM[1:0] (409M params) reached 13.43 validation perplexity and xLSTM[7:1] (408M) 13.48, outperforming Mamba (423M, 13.70), Llama (407M, 14.25), GPT-3 (356M, 14.26), and other compared methods.
  • On 300B SlimPajama tokens at 1.3B scale, xLSTM[1:0] reached 8.89 validation perplexity and xLSTM[7:1] 9.00, versus Mamba 9.14, Llama 9.44, and RWKV-4 9.83; at 16k context, xLSTM[7:1] reached 8.92 and xLSTM[1:0] 9.01, versus Mamba 14.00, RWKV-4 13.75, and Llama 337.83.
  • On PALOMA, xLSTM[1:0] had lower perplexity than Mamba in 568/571 (99.5%) text domains, than Llama in 486/571 (85.1%), and than RWKV-4 in 570/571 (99.8%).
  • Ablations attribute gains to both exponential gating and matrix memory: vanilla multi-layer LSTM had 2417.86 perplexity, adding ResNet backbone 35.46, up-projection backbone 26.01, exponential gating (xLSTM[0:1]) 17.70, and matrix memory (xLSTM[7:1]) 13.48.

Methods

  • The paper introduces sLSTM with scalar memory, scalar update, exponential gating, normalizer/stabilizer states, and new memory mixing; and mLSTM with matrix memory, covariance (outer product) update rule, normalizer state, and full parallelizability.
  • These are integrated into residual xLSTM blocks (post up-projection for sLSTM, pre up-projection for mLSTM) and residually stacked into xLSTM architectures; notation xLSTM[a:b] gives the ratio of mLSTM- to sLSTM-based blocks.
  • Experiments include synthetic formal-language/state-tracking tasks, Multi-Query Associative Recall, Long Range Arena, and language modeling on SlimPajama with 15B and 300B tokens; main 300B comparisons use 125M, 350M, 760M, and 1.3B model sizes against RWKV-4, Llama, and Mamba.

Limitations

  • The paper states: sLSTM memory mixing prevents parallelizable operations, though a fast CUDA kernel is less than two times slower than parallel mLSTM; mLSTM CUDA kernels are not optimized and are about 4 times slower than FlashAttention or Mamba’s scan; mLSTM’s d×d matrix memory has high computation complexity; forget-gate initialization must be chosen carefully; the matrix memory is independent of sequence length and may overload for longer contexts, though not up to 16k; and the architecture/hyperparameters were not fully optimized, especially for larger xLSTM models.
  • Evident from the text: the main 300B-token LLM comparison excludes RWKV-5, RWKV-6, and HGRN2 because reasonable training precision was found only after the 300B experiments began; the largest main 300B models are 1.3B, with scaling-law plots extending to 2.7B in the appendix.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.52202/079017-3417

Cites

2 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Figures

Figure 1 The extended LSTM (xLSTM) family. From left to right: 1. The original LSTM memory cell with constant error carousel and gating. 2. New sLSTM and mLSTM memory cells that introduce exponential gating. sLSTM offers a new memory mixing technique. mLSTM is fully parallelizable with a novel matrix memory cell state and new covariance update rule. 3. mLSTM and sLSTM in residual blocks yield xLSTM blocks. 4. Stacked xLSTM blocks give an xLSTM architecture.
Figure 1 The extended LSTM (xLSTM) family. From left to right: 1. The original LSTM memory cell with constant error carousel and gating. 2. New sLSTM and mLSTM memory cells that introduce exponential gating. sLSTM offers a new memory mixing technique. mLSTM is fully parallelizable with a novel matrix memory cell state and new covariance update rule. 3. mLSTM and sLSTM in residual blocks yield xLSTM blocks. 4. Stacked xLSTM blocks give an xLSTM architecture.CC-BY · From the PDF
Figure 2 LSTM limitations. Left : Nearest Neighbor Search problem in terms of mean squared error (MSE). Given a reference vector, a sequence is scanned sequentially for the most similar vector with the objective to return its attached value at sequence end. LSTM struggles to revise a stored value when a more similar vector is found. Our new xLSTM overcomes this limitation by exponential gating. Right : Rare Token Prediction. The perplexity (PPL) of token prediction on Wikitext-103, in partitions of token frequency. LSTM performs worse on predicting rare tokens because of its limited storage capacities, whereas our new xLSTM solves this problem via a matrix memory.
Figure 2 LSTM limitations. Left : Nearest Neighbor Search problem in terms of mean squared error (MSE). Given a reference vector, a sequence is scanned sequentially for the most similar vector with the objective to return its attached value at sequence end. LSTM struggles to revise a stored value when a more similar vector is found. Our new xLSTM overcomes this limitation by exponential gating. Right : Rare Token Prediction. The perplexity (PPL) of token prediction on Wikitext-103, in partitions of token frequency. LSTM performs worse on predicting rare tokens because of its limited storage capacities, whereas our new xLSTM solves this problem via a matrix memory.CC-BY · From the PDF
Figure 3 xLSTM blocks. Left : A residual sLSTM block with post up-projection (like Transformers): The input is fed into an sLSTM - with an optional convolution - followed by a gated MLP. Right : A residual mLSTM block with pre up-projection (like State Space models): mLSTM is wrapped inside two MLPs, via a convolution, a learnable skip connection, and an output gate that acts componentwise. See Figure 10 and Figure 11 in the appendix for details.
Figure 3 xLSTM blocks. Left : A residual sLSTM block with post up-projection (like Transformers): The input is fed into an sLSTM - with an optional convolution - followed by a gated MLP. Right : A residual mLSTM block with pre up-projection (like State Space models): mLSTM is wrapped inside two MLPs, via a convolution, a learnable skip connection, and an output gate that acts componentwise. See Figure 10 and Figure 11 in the appendix for details.CC-BY · From the PDF
Figure 5 Test of memory capacities of different models at the Multi-Query Associative Recall task with context length 2048. Each panel is dedicated to a different number of key-value pairs. The x -axis displays the model size and the y -axis the validation accuracy.
Figure 5 Test of memory capacities of different models at the Multi-Query Associative Recall task with context length 2048. Each panel is dedicated to a different number of key-value pairs. The x -axis displays the model size and the y -axis the validation accuracy.CC-BY · From the PDF
Figure 6 Method comparison on next token prediction when trained on 15B tokens from SlimPajama. Performance measure in validation perplexity for the best methods of each model class (see Table 1) are reported. The performance degradation of xLSTM[7:1] at 2.7B is due to initially slower training convergence that leads to an especially undertrained model. xLSTM is the best method at all sizes.
Figure 6 Method comparison on next token prediction when trained on 15B tokens from SlimPajama. Performance measure in validation perplexity for the best methods of each model class (see Table 1) are reported. The performance degradation of xLSTM[7:1] at 2.7B is due to initially slower training convergence that leads to an especially undertrained model. xLSTM is the best method at all sizes.CC-BY · From the PDF
Figure 7 Sequence extrapolation in language modeling. This is a comparison of 1.3B-sized, large models of xLSTM, RWKV-4, Llama, and Mamba at next token prediction on the SlimPajama validation set after training on 300B tokens from SlimPajama. Models are trained with context length 2048 and then tested for context lengths up to 16384. Left: Token perplexities evaluated at different context lengths. In contrast to other methods, xLSTM models remain at low perplexities for longer contexts. Right: Prediction quality when extrapolating to long context sizes in terms of validation perplexity (PPL). xLSTM yields the best PPL values (best in bold, second best underlined).
Figure 7 Sequence extrapolation in language modeling. This is a comparison of 1.3B-sized, large models of xLSTM, RWKV-4, Llama, and Mamba at next token prediction on the SlimPajama validation set after training on 300B tokens from SlimPajama. Models are trained with context length 2048 and then tested for context lengths up to 16384. Left: Token perplexities evaluated at different context lengths. In contrast to other methods, xLSTM models remain at low perplexities for longer contexts. Right: Prediction quality when extrapolating to long context sizes in terms of validation perplexity (PPL). xLSTM yields the best PPL values (best in bold, second best underlined).CC-BY · From the PDF
Image 7
CC-BY · From the PDF
Figure 9 Inference Generative Speed. Left: Generation times of different 1.3B models for a pre-fill context of 16 tokens (to mitigate cache initialization). The recurrent models (xLSTM[1:0], xLSTM[7:1], Mamba and RWKV-4) show linear behavior, whereas the Transformer (Llama) inference/decoding time is quadratic in sequence length. Right: Token throughput for different batch sizes on a A100-80GB GPU for 1.3B sized models. Note that the Transformer / Llama model goes out of memory (OOM) already for small batch sizes, whereas xLSTM and Mamba can sustain very large batch sizes. xLSTM[1:0] consistently outperforms Mamba in throughput.
Figure 9 Inference Generative Speed. Left: Generation times of different 1.3B models for a pre-fill context of 16 tokens (to mitigate cache initialization). The recurrent models (xLSTM[1:0], xLSTM[7:1], Mamba and RWKV-4) show linear behavior, whereas the Transformer (Llama) inference/decoding time is quadratic in sequence length. Right: Token throughput for different batch sizes on a A100-80GB GPU for 1.3B sized models. Note that the Transformer / Llama model goes out of memory (OOM) already for small batch sizes, whereas xLSTM and Mamba can sustain very large batch sizes. xLSTM[1:0] consistently outperforms Mamba in throughput.CC-BY · From the PDF
Figure 10 Schematic representation of an sLSTM Block - post up-projection: Embedded in a preLayerNorm residual structure, the input is optionally passed through a causal convolution of window size 4 that includes a Swish activation for input and forget gates. Then, for all input, forget and output gates i , f , o , and the cell update z the input is fed through a block-diagonal linear layer with four diagonal blocks or 'heads'. These diagonal blocks coincide with the recurrent gate pre-activations from the last hidden state, which corresponds to an sLSTM with four heads depicted with the circular arrows. The resulting hidden state goes through a GroupNorm layer (Wu & He, 2018) - a head-wise LayerNorm for each of the four heads. Finally, the output is up- and down-projected using a gated MLP, with GeLU activation function and projection factor 4 / 3 to match parameters.
Figure 10 Schematic representation of an sLSTM Block - post up-projection: Embedded in a preLayerNorm residual structure, the input is optionally passed through a causal convolution of window size 4 that includes a Swish activation for input and forget gates. Then, for all input, forget and output gates i , f , o , and the cell update z the input is fed through a block-diagonal linear layer with four diagonal blocks or 'heads'. These diagonal blocks coincide with the recurrent gate pre-activations from the last hidden state, which corresponds to an sLSTM with four heads depicted with the circular arrows. The resulting hidden state goes through a GroupNorm layer (Wu & He, 2018) - a head-wise LayerNorm for each of the four heads. Finally, the output is up- and down-projected using a gated MLP, with GeLU activation function and projection factor 4 / 3 to match parameters.CC-BY · From the PDF
Figure 11 Schematic representation of an mLSTM block - pre up-projection: Embedded in a pre-LayerNorm residual structure, the input is up-projected first with projection factor 2, once for an externalized output gate and once as input for the mLSTM cells. The mLSTM cell input is dimension-wise causally convoluted (kernel size 4), before entering a learnable skip connection. We obtain input q and k via block-diagonal projection matrices of block size 4. The values v are fed directly, skipping the convolution part. After the mLSTM sequence mixing, outputs are normalized via GroupNorm (Wu & He, 2018) - a head-wise layer norm for each of the four heads. Finally, the learnable skip input is added and the result is gated component-wise with the external output gate. The output is down-projected.
Figure 11 Schematic representation of an mLSTM block - pre up-projection: Embedded in a pre-LayerNorm residual structure, the input is up-projected first with projection factor 2, once for an externalized output gate and once as input for the mLSTM cells. The mLSTM cell input is dimension-wise causally convoluted (kernel size 4), before entering a learnable skip connection. We obtain input q and k via block-diagonal projection matrices of block size 4. The values v are fed directly, skipping the convolution part. After the mLSTM sequence mixing, outputs are normalized via GroupNorm (Wu & He, 2018) - a head-wise layer norm for each of the four heads. Finally, the learnable skip input is added and the result is gated component-wise with the external output gate. The output is down-projected.CC-BY · From the PDF
Image 11
CC-BY · From the PDF
Figure 14 Result of MQAR-Experiment 1. The columns show different task settings (context length and key-value pairs). The rows group related models for better clarity. The x -axis gives the model size and the y -axis the validation accuracy.
Figure 14 Result of MQAR-Experiment 1. The columns show different task settings (context length and key-value pairs). The rows group related models for better clarity. The x -axis gives the model size and the y -axis the validation accuracy.CC-BY · From the PDF
Figure 15 Result of MQAR-Experiment 2. The columns and rows correspond to different numbers of key-value pairs and the context length respectivly. The x -axis gives the model size and the y -axis the validation accuracy.
Figure 15 Result of MQAR-Experiment 2. The columns and rows correspond to different numbers of key-value pairs and the context length respectivly. The x -axis gives the model size and the y -axis the validation accuracy.CC-BY · From the PDF
Figure 16 Result of MQAR-Experiment 3 (Extrapolation). All evaluated models were trained on context length 2048 and the number of key-value pairs given by the columns of the plot. The rows show the different context lengths used in the evaluation. The x -axis gives the model size and the y -axis the validation accuracy.
Figure 16 Result of MQAR-Experiment 3 (Extrapolation). All evaluated models were trained on context length 2048 and the number of key-value pairs given by the columns of the plot. The rows show the different context lengths used in the evaluation. The x -axis gives the model size and the y -axis the validation accuracy.CC-BY · From the PDF

Verification

Claims the paper makes

10 Valid · 4 Not valid · 9 Not checked · 95 Own results and interpretations

59 more claims

Create a free account to see all 118 claims and every source verdict.

See all claims — free account
  • Probably not a substantial claim. 30
  • It may state two separate claims as one. 33
  • The quote only partly states this claim. 1

Create a free account to see these claims.

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

3 of 3

Citations

Cites

The publisher hasn’t deposited this paper’s reference list.

Part of this paper’s reference list could not be split into single entries.

Research works printed but not linked: 18

  1. J. Ba, G. E. Hinton, V. Mnih, J. Z. Leibo, and C. Ionescu. Using fast weights to attend to the recent past. In D. D. Lee, M. Sugiyama, U. V. Luxburg, I. Guyon, and R. Garnett (eds.), Advances in Neural Information Processing Systems 29, pp. 4331-4339.
  2. T. Brown, B. Mann, N. Ryder, et al. Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 1877-1901.
  3. T. M. Cover. Geometrical and statistical properties of systems of linear inequalities with applications in pattern recognition. Electronic Computers, IEEE Transactions on, EC-14(3):326-334, 1965.
  4. T. Dao. Flashattention-2: Faster attention with better parallelism and work partitioning. In In- ternational Conference on Learning Representations (ICLR), volume 12, 2024. URL https: //openreview.net/forum?id=mZn2Xyh9Ec.
  5. S. Hochreiter. Untersuchungen zu dynamischen neuronalen Netzen. Master's thesis, Technische Universität München, 1991.
  6. S. Hochreiter and J. Schmidhuber. LSTM can solve hard long time lag problems. In M. C. Mozer, M. I. Jordan, and T. Petsche (eds.), Advances in Neural Information Processing Systems (NeurIPS), volume 9, pp. 473-479. MIT Press, Cambridge MA, 1997b.
  7. S. Hochreiter, Y. Bengio, P. Frasconi, and J. Schmidhuber. Gradient flow in recurrent nets: the difficulty of learning long-term dependencies. In J. Kolen and S. Kremer (eds.), A Field Guide to Dynamical Recurrent Networks. IEEE, 2000.
  8. A. Krizhevsky. Learning multiple layers of features from tiny images. Master's thesis, Deptartment of Computer Science, University of Toronto, 2009.
  9. D. Krotov and J. J. Hopfield. Dense associative memory for pattern recognition. In D. D. Lee, M. Sugiyama, U. V. Luxburg, I. Guyon, and R. Garnett (eds.), Advances in Neural Information Processing Systems, pp. 1172-1180.
  10. I. Loshchilov and F. Hutter. Decoupled weight decay regularization. In International Confer- ence on Learning Representations (ICLR), 2019. URL https://openreview.net/forum?id= Bkg6RiCqY7.
  11. A. L. Maas, R. E. Daly, P. T. Pham, D. Huang, A. Y. Ng, and C. Potts. Learning word vectors for sentiment analysis. In Annual Meeting of the Association for Computational Linguistics, volume 49, pp. 142-150, 2011.
  12. I. Schlag, K. Irie, and J. Schmidhuber. Linear transformers are secretly fast weight programmers. In M. Meila and T. Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning (ICML), volume 139 of Proceedings of Machine Learning Research, pp. 9355-9366.
  13. PMLR, 2021.
  14. J. Schmidhuber. Learning to control fast-weight memories: An alternative to recurrent nets. Neural Computation, 4(1):131-139, 1992.
  15. L. Soldaini, R. Kinney, A. Bhagia, et al. Dolma: an open corpus of three trillion tokens for language model pretraining research. ArXiv, 2306.01116, 2023. · arXiv
  16. R. K. Srivastava, K. Greff, and J. Schmidhuber. Training very deep networks. In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett (eds.), Advances in Neural Information Processing Systems (NeurIPS), volume 28.
  17. I. Sutskever, O. Vinyals, and Q. V. V. Le. Sequence to sequence learning with neural networks. In Z. Ghahramani, M. Welling, C. Cortes, N. D. Lawrence, and K. Q. Weinberger (eds.), Advances in Neural Information Processing Systems 27 (NIPS'13), pp. 3104-3112.
  18. A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems (NeurIPS), volume 30, pp. 5998-6008.

Cited by

See all 286

Metadata

Authors
Beck, M., Pöppel, K., Spanring, M., Auer, A., Prudnikova, O., Kopp, M., Klambauer, G., Brandstetter, J., & Hochreiter, S.
Venue
Advances in Neural Information Processing Systems 37
Published
2024
Pages
107547-107603
ISSN
1049-5258 (ISSN-L)
OpenAlex
W4415795940
Cited by
286
References
0
Citations by year
2026: 137 · 2025: 126 · 2024: 23
FWCI
83.96
Type
conference paper
Language
en
Open access
Licence
cc-by
Locations
  • neural information processing systems (publishedVersion)
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Neural Networks and Applications
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
Keywords
long short-term memory, recurrent neural networks, xLSTM, memory expansion, sequence modeling

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-29, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-29 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol and qwen3.8-omni-flash).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: xLSTM: Extended Long Short-Term Memory. https://app.cochat.ai/papers/10.52202/079017-3417
The paper
Beck, M., Pöppel, K., Spanring, M., Auer, A., Prudnikova, O., Kopp, M., Klambauer, G., Brandstetter, J., & Hochreiter, S. (2024). xLSTM: Extended Long Short-Term Memory. Advances in Neural Information Processing Systems 37. https://doi.org/10.52202/079017-3417

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.