The DOI matches its Crossref record; 99 of the 101 references it cites were found in a registry or an index; 19 of the 35 sources its claims cite were checked for support.
xLSTM: Extended Long Short-Term Memory
Open accessAbstract
In the 1990s, the constant error carousel and gating were introduced as the central ideas of the Long Short-Term Memory (LSTM). Since then, LSTMs have stood the test of time and contributed to numerous deep learning success stories, in particular they constituted the first Large Language Models (LLMs). However, the advent of the Transformer technology with parallelizable self-attention at its core marked the dawn of a new era, outpacing LSTMs at scale. We now raise a simple question: How far do we get in language modeling when scaling LSTMs to billions of parameters, leveraging the latest techniques from modern LLMs, but mitigating known limitations of LSTMs? Firstly, we introduce exponential gating with appropriate normalization and stabilization techniques. Secondly, we modify the LSTM memory structure, obtaining: (i) sLSTM with a scalar memory, a scalar update, and new memory mixing, (ii) mLSTM that is fully parallelizable with a matrix memory and a covariance update rule. Integrating these LSTM extensions into residual block backbones yields xLSTM blocks that are then residually stacked into xLSTM architectures. Exponential gating and modified memory structures boost xLSTM capabilities to perform favorably when compared to state-of-the-art Transformers and State Space Models, both in performance and scaling.
Overview
CoChat Summary
TL;DR
xLSTM extends LSTM with exponential gating and new memory structures—sLSTM (scalar memory, memory mixing) and mLSTM (matrix memory, covariance update, parallelizable)—and shows favorable language-modeling performance versus Transformers and State Space Models. On 15B and 300B SlimPajama tokens, xLSTM variants achieve the best validation perplexity among compared methods and extrapolate better to longer contexts.
Key findings
- On 15B SlimPajama tokens, xLSTM[1:0] (409M params) reached 13.43 validation perplexity and xLSTM[7:1] (408M) 13.48, outperforming Mamba (423M, 13.70), Llama (407M, 14.25), GPT-3 (356M, 14.26), and other compared methods.
- On 300B SlimPajama tokens at 1.3B scale, xLSTM[1:0] reached 8.89 validation perplexity and xLSTM[7:1] 9.00, versus Mamba 9.14, Llama 9.44, and RWKV-4 9.83; at 16k context, xLSTM[7:1] reached 8.92 and xLSTM[1:0] 9.01, versus Mamba 14.00, RWKV-4 13.75, and Llama 337.83.
- On PALOMA, xLSTM[1:0] had lower perplexity than Mamba in 568/571 (99.5%) text domains, than Llama in 486/571 (85.1%), and than RWKV-4 in 570/571 (99.8%).
- Ablations attribute gains to both exponential gating and matrix memory: vanilla multi-layer LSTM had 2417.86 perplexity, adding ResNet backbone 35.46, up-projection backbone 26.01, exponential gating (xLSTM[0:1]) 17.70, and matrix memory (xLSTM[7:1]) 13.48.
Methods
- The paper introduces sLSTM with scalar memory, scalar update, exponential gating, normalizer/stabilizer states, and new memory mixing; and mLSTM with matrix memory, covariance (outer product) update rule, normalizer state, and full parallelizability.
- These are integrated into residual xLSTM blocks (post up-projection for sLSTM, pre up-projection for mLSTM) and residually stacked into xLSTM architectures; notation xLSTM[a:b] gives the ratio of mLSTM- to sLSTM-based blocks.
- Experiments include synthetic formal-language/state-tracking tasks, Multi-Query Associative Recall, Long Range Arena, and language modeling on SlimPajama with 15B and 300B tokens; main 300B comparisons use 125M, 350M, 760M, and 1.3B model sizes against RWKV-4, Llama, and Mamba.
Limitations
- The paper states: sLSTM memory mixing prevents parallelizable operations, though a fast CUDA kernel is less than two times slower than parallel mLSTM; mLSTM CUDA kernels are not optimized and are about 4 times slower than FlashAttention or Mamba’s scan; mLSTM’s d×d matrix memory has high computation complexity; forget-gate initialization must be chosen carefully; the matrix memory is independent of sequence length and may overload for longer contexts, though not up to 16k; and the architecture/hyperparameters were not fully optimized, especially for larger xLSTM models.
- Evident from the text: the main 300B-token LLM comparison excludes RWKV-5, RWKV-6, and HGRN2 because reasonable training precision was found only after the 300B experiments began; the largest main 300B models are 1.3B, with scaling-law plots extending to 2.7B in the appendix.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures




![Figure 6 Method comparison on next token prediction when trained on 15B tokens from SlimPajama. Performance measure in validation perplexity for the best methods of each model class (see Table 1) are reported. The performance degradation of xLSTM[7:1] at 2.7B is due to initially slower training convergence that leads to an especially undertrained model. xLSTM is the best method at all sizes.](https://figures.cochat.ai/paper-figures/1c48bfb626c14c6795ef465de3ead032/xlstm-extended-long-short-term-memory-figure-6.png)


![Figure 9 Inference Generative Speed. Left: Generation times of different 1.3B models for a pre-fill context of 16 tokens (to mitigate cache initialization). The recurrent models (xLSTM[1:0], xLSTM[7:1], Mamba and RWKV-4) show linear behavior, whereas the Transformer (Llama) inference/decoding time is quadratic in sequence length. Right: Token throughput for different batch sizes on a A100-80GB GPU for 1.3B sized models. Note that the Transformer / Llama model goes out of memory (OOM) already for small batch sizes, whereas xLSTM and Mamba can sustain very large batch sizes. xLSTM[1:0] consistently outperforms Mamba in throughput.](https://figures.cochat.ai/paper-figures/f63789279b6e4ff4ba34314a9b59cd7d/xlstm-extended-long-short-term-memory-figure-9.png)






Verification
Claims the paper makes
10 Valid · 4 Not valid · 9 Not checked · 95 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“This result is in agreement with findings that Transformers and State Space models are fundamentally less powerful than RNNs (Merrill et al., 2024;Merrill & Sabharwal, 2023;Delétang et al., 2023).”
§ xLSTM: Extended Long Short-Term Memory > 4.1 Synthetic Tasks and Long Range Arena
What the sources say
- Partly supportedcites Merrill et al., 2024 sourceExperimental — accuracy not yet measured
The Introduction says that linear and Mamba-style SSMs, like transformers, cannot express certain sequential state-tracking problems that RNNs can. But the claim is broader than the paper’s result: the Discussion (§4.4) limits the separation to specified SSM architectures and assumptions, and the Conclusion (§7) describes an input-dependent SSM that can express the S5 word problem. (LLM, conf=0.87)
- Partly supportedcites Merrill & Sabharwal, 2023 sourceExperimental — accuracy not yet measured
The Abstract and §2 support a computational limitation for log-precision Transformers, linking it to their parallelizability. They do not establish that State Space models are less powerful than RNNs, or make that comparison for Transformers; the paper’s result is bounded by its precision and model assumptions. The regular-grammar result is the citing authors’ own framing, not a finding attributed to this source. (LLM, conf=0.86)
- Not matchedcites Delétang et al., 2023
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Especially its quality of in-context learning seems to be strongly connected to this capability (Olsson et al., 2022).”
§ xLSTM: Extended Long Short-Term Memory > Why Multi-Query Associative Recall for Memory Tests of LLM Architectures.
What the sources say
- Supportedcites Olsson et al., 2022 source
The Induction Heads section (p. 5) describes retrieving a token associated with an earlier occurrence of the current token—the capability described here as associative recall. Assessing the Evidence (p. 15) reports that the formation of these heads correlates with a large increase in in-context-learning capacity, while cautioning that causation is not established. The claim says the capabilities seem connected, not that one causes the other. (LLM, conf=0.85)
boostedstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“In accordance to the results of Arora et al. (2023) H3, Hyena, RWKV-4 fail to solve the task with a smaller model dimension.”
§ xLSTM: Extended Long Short-Term Memory > Why Multi-Query Associative Recall for Memory Tests of LLM Architectures.
What the sources say
- Partly supportedcites Arora et al. (2023) sourceExperimental — accuracy not yet measured
The Setup section reports that the tested gated-convolution architectures—H3, Hyena, and RWKV—did not reach MQAR accuracy above 0.9 unless model dimension was at least the sequence length. Our copy calls the tested model RWKV, not specifically RWKV-4, so it does not establish the version named in the claim. (LLM, conf=0.82)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Associative Recall (AR), the ability to retrieve a specific value (information) associated with a given key (information), constitutes a key capability for LLM to perform well (Poli et al., 2024;Arora et al., 2023;Olsson et al., 2022).”
§ xLSTM: Extended Long Short-Term Memory > Why Multi-Query Associative Recall for Memory Tests of LLM Architectures.
What the sources say
- Supportedcites Poli et al., 2024 source
Section 3.1.1 'In-context recall' states that 'To answer a prompt well, language models must be able to understand and learn from new information presented in the prompt', and that 'the associative recall task ... is well-suited to test a specific subset of in-context learning ability: direct lookup', defined as 'Given an input sequence of key-value pairs, models are tasked with retrieving all values from the input sequence associated with keys that were already shown in the input sequence.' Appendix B.1.1 repeats the key–value retrieval definition. The claim's definition of AR and its framing as a capability required for good prompt-level performance therefore match what our copy attributes, with the minor nuance that the paper treats AR as a probe of a subset (direct lookup) of in-context learning ability. (LLM, conf=0.85)
- Supportedcites Arora et al., 2023 source
Section 3.1 defines associative recall as recalling an association previously seen in context, illustrated by retrieving “Rice” from “Tim.” The Introduction says this ability is predictive of in-context learning quality, and Section 3 reports that it accounts for 82% of the measured perplexity gap between the model architectures studied. Together, these support the claim that associative recall is an important capability for language-model performance. (LLM, conf=0.91)
- Partly supportedcites Olsson et al., 2022 sourceExperimental — accuracy not yet measured
The Induction Heads section (p. 5) describes retrieving a token B previously associated with token A, matching the claim’s description of associative recall. The paper presents the broader claim that this mechanism plays a major role in in-context learning as a preliminary hypothesis (p. 2); it does not establish associative recall as a general requirement for LLMs to perform well. (LLM, conf=0.82)
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“The results show that, in contrast to other methods (Katharopoulos et al., 2020;Sun et al., 2023;Qin et al., 2023;Katsch, 2023;Yang et al., 2023;Qin et al., 2024;Peng et al., 2024), having the gates both learnable and input dependent gives the best results.”
§ xLSTM: Extended Long Short-Term Memory > Additional information
What the sources say
- Partly supportedcites Katharopoulos et al., 2020 sourceExperimental — accuracy not yet measured
Section 5 (p. 8) identifies this paper’s method as a linear transformer, so it can serve as an example of an ‘other method.’ Its reported results compare linear attention with other transformer approaches, not learnable, input-dependent gates or mLSTM. The claimed best results for mLSTM are therefore not supported by this source. (LLM, conf=0.78)
- Supportedcites Sun et al., 2023 source
The citation identifies RetNet as a contrasting method, rather than attributing the mLSTM results to it. Section 2.2, “Gated Multi-Scale Retention” (page unavailable), says RetNet keeps its decay rates fixed, even though it also uses a learned, input-dependent swish gate. The paper does not report the mLSTM result; that result is presented as the citing authors’ own finding. (LLM, conf=0.85)
- Partly supportedcites Qin et al., 2023 sourceExperimental — accuracy not yet measured
The Results (§4.2, page unavailable) report that HGRN performs worse with a data-independent forget gate and benefits from combining gating with a learnable lower bound. That supports the direction of the gate comparison, but these are HGRN results, not a finding that learnable, input-dependent gates give the best results for mLSTM. (LLM, conf=0.87)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
59 more claims
Create a free account to see all 118 claims and every source verdict.
- Probably not a substantial claim. 30
- It may state two separate claims as one. 33
- The quote only partly states this claim. 1
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ xLSTM: Extended Long Short-Term Memory > General Details on Ablation Studies.
“Mamba 793.2 182.74 22.58 10.47 36.25”
§ xLSTM: Extended Long Short-Term Memory > 4.3 xLSTM as Large Language Model
“Mamba 870.5 10.24 9.24 50.84 53.97 71.16 60.44 29.78 56.99 53.86”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ xLSTM: Extended Long Short-Term Memory > 4.3 xLSTM as Large Language Model
“Model sizes are 125M, 250M, 760M, and 1.3B.”
§ xLSTM: Extended Long Short-Term Memory > 4.3 xLSTM as Large Language Model
“We train different model sizes (125M, 350M, 760M, 1.3B)”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ xLSTM: Extended Long Short-Term Memory > A.2 sLSTM
“δ b = sign (n) ⊙ δ n ⊙ 1 if |b| > exp(-m) 0 otherwise (90”
§ xLSTM: Extended Long Short-Term Memory > A.2 sLSTM
“n = max (|b|, exp(-m))(84)”
Citations
Cites
The publisher hasn’t deposited this paper’s reference list.
Part of this paper’s reference list could not be split into single entries.
Research works printed but not linked: 18
- J. Ba, G. E. Hinton, V. Mnih, J. Z. Leibo, and C. Ionescu. Using fast weights to attend to the recent past. In D. D. Lee, M. Sugiyama, U. V. Luxburg, I. Guyon, and R. Garnett (eds.), Advances in Neural Information Processing Systems 29, pp. 4331-4339.
- T. Brown, B. Mann, N. Ryder, et al. Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 1877-1901.
- T. M. Cover. Geometrical and statistical properties of systems of linear inequalities with applications in pattern recognition. Electronic Computers, IEEE Transactions on, EC-14(3):326-334, 1965.
- T. Dao. Flashattention-2: Faster attention with better parallelism and work partitioning. In In- ternational Conference on Learning Representations (ICLR), volume 12, 2024. URL https: //openreview.net/forum?id=mZn2Xyh9Ec.
- S. Hochreiter. Untersuchungen zu dynamischen neuronalen Netzen. Master's thesis, Technische Universität München, 1991.
- S. Hochreiter and J. Schmidhuber. LSTM can solve hard long time lag problems. In M. C. Mozer, M. I. Jordan, and T. Petsche (eds.), Advances in Neural Information Processing Systems (NeurIPS), volume 9, pp. 473-479. MIT Press, Cambridge MA, 1997b.
- S. Hochreiter, Y. Bengio, P. Frasconi, and J. Schmidhuber. Gradient flow in recurrent nets: the difficulty of learning long-term dependencies. In J. Kolen and S. Kremer (eds.), A Field Guide to Dynamical Recurrent Networks. IEEE, 2000.
- A. Krizhevsky. Learning multiple layers of features from tiny images. Master's thesis, Deptartment of Computer Science, University of Toronto, 2009.
- D. Krotov and J. J. Hopfield. Dense associative memory for pattern recognition. In D. D. Lee, M. Sugiyama, U. V. Luxburg, I. Guyon, and R. Garnett (eds.), Advances in Neural Information Processing Systems, pp. 1172-1180.
- I. Loshchilov and F. Hutter. Decoupled weight decay regularization. In International Confer- ence on Learning Representations (ICLR), 2019. URL https://openreview.net/forum?id= Bkg6RiCqY7.
- A. L. Maas, R. E. Daly, P. T. Pham, D. Huang, A. Y. Ng, and C. Potts. Learning word vectors for sentiment analysis. In Annual Meeting of the Association for Computational Linguistics, volume 49, pp. 142-150, 2011.
- I. Schlag, K. Irie, and J. Schmidhuber. Linear transformers are secretly fast weight programmers. In M. Meila and T. Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning (ICML), volume 139 of Proceedings of Machine Learning Research, pp. 9355-9366.
- PMLR, 2021.
- J. Schmidhuber. Learning to control fast-weight memories: An alternative to recurrent nets. Neural Computation, 4(1):131-139, 1992.
- L. Soldaini, R. Kinney, A. Bhagia, et al. Dolma: an open corpus of three trillion tokens for language model pretraining research. ArXiv, 2306.01116, 2023. · arXiv
- R. K. Srivastava, K. Greff, and J. Schmidhuber. Training very deep networks. In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett (eds.), Advances in Neural Information Processing Systems (NeurIPS), volume 28.
- I. Sutskever, O. Vinyals, and Q. V. V. Le. Sequence to sequence learning with neural networks. In Z. Ghahramani, M. Welling, C. Cortes, N. D. Lawrence, and K. Q. Weinberger (eds.), Advances in Neural Information Processing Systems 27 (NIPS'13), pp. 3104-3112.
- A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems (NeurIPS), volume 30, pp. 5998-6008.
Cited by
RULER: What's the Real Context Size of Your Long-Context Language Models?
Hsieh, C.-P., Sun, S., Kriman, S., Acharya, S., Rekesh, D., Jia, F., Zhang, Y., & Ginsburg, B. (2024) · arXiv · doi:10.48550/arxiv.2404.06654 · Cited by 1,094
A review of large language models and autonomous agents in chemistry
Ramos, M. C., Collison, C. J., & White, A. D. (2025) · Chemical Science · doi:10.1039/d4sc03921a · Cited by 274
Wang, F., Zhang, Z., Zhang, X., Wu, Z., Mo, T., Lu, Q., Wang, W., Li, R., Xu, J., Tang, X., et al. (2025) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3768165 · Cited by 106
Remaining useful life prediction of PEMFC based on matrix long short-term memory
Yi, F., Shu, X., Zhou, J., Zhang, J., Feng, C., Gong, H., Zhang, C., & Yu, W. (2025) · International Journal of Hydrogen Energy · doi:10.1016/j.ijhydene.2025.02.302 · Cited by 73
Nested Learning: The Illusion of Deep Learning Architectures
Behrouz, A., Razaviyayn, M., Zhong, P., & Mirrokni, V. (2025) · arXiv · doi:10.48550/arxiv.2512.24695 · Cited by 37
Metadata
- Authors
- Beck, M., Pöppel, K., Spanring, M., Auer, A., Prudnikova, O., Kopp, M., Klambauer, G., Brandstetter, J., & Hochreiter, S.
- ORCID
- Venue
- Advances in Neural Information Processing Systems 37
- Published
- 2024
- Pages
- 107547-107603
- ISSN
- 1049-5258 (ISSN-L)
- OpenAlex
- W4415795940
- Cited by
- 286
- References
- 0
- Citations by year
- 2026: 137 · 2025: 126 · 2024: 23
- FWCI
- 83.96
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- cc-by
- Locations
- neural information processing systems (publishedVersion)
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Neural Networks and Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
- Keywords
- long short-term memory, recurrent neural networks, xLSTM, memory expansion, sequence modeling
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-29, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-29 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol and qwen3.8-omni-flash).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: xLSTM: Extended Long Short-Term Memory. https://app.cochat.ai/papers/10.52202/079017-3417
- The paper
- Beck, M., Pöppel, K., Spanring, M., Auer, A., Prudnikova, O., Kopp, M., Klambauer, G., Brandstetter, J., & Hochreiter, S. (2024). xLSTM: Extended Long Short-Term Memory. Advances in Neural Information Processing Systems 37. https://doi.org/10.52202/079017-3417
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.