Recurrent Memory Transformer
Open accessAbstract
Transformer-based models show their effectiveness across multiple domains and tasks. The self-attention allows to combine information from all sequence elements into context-aware representations. However, global and local information has to be stored mostly in the same element-wise representations. Moreover, the length of an input sequence is limited by quadratic computational complexity of self-attention. In this work, we propose and study a memory-augmented segment-level recurrent Transformer (RMT). Memory allows to store and process local and global information as well as to pass information between segments of the long sequence with the help of recurrence. We implement a memory mechanism with no changes to Transformer model by adding special memory tokens to the input or output sequence. Then the model is trained to control both memory operations and sequence representations processing. Results of experiments show that RMT performs on par with the Transformer-XL on language modeling for smaller memory sizes and outperforms it for tasks that require longer sequence processing. We show that adding memory tokens to Tr-XL is able to improve its performance. This makes Recurrent Memory Transformer a promising architecture for applications that require learning of long-term dependencies and general purpose in memory processing, such as algorithmic tasks and reasoning.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
LLMs are fundamentally limited by their reliance on fixed context windows, which severely restrict their ability to maintain coherence over extended interactions.
“However, these systems are fundamentally limited by their reliance on fixed context windows, which severely restrict their ability to maintain coherence over extended interactions (Bulatov et al., 2022, Liu et al., 2023).”
From Mem0: Building Production-Ready AI Agents with Scalable Long-Term Memory (2025) · cites this paper as Bulatov et al., 2022
The Abstract (p. 1) says Transformer input length is limited by the quadratic cost of self-attention and presents recurrent memory as a way to pass information between long-sequence segments. The Introduction (p. 1) likewise says poor scaling hurts applications to long inputs. These passages support a limitation in processing extended context, but the paper does not establish that fixed context windows severely restrict coherence over extended interactions. (LLM, conf=0.86)
Experimental — accuracy not yet measuredSparse retrieval, compressed session vectors, memory-native architectures like Recurrent Memory Transformers, and retrieval-free injection via adapters all point toward cheaper alternatives.
“Sparse retrieval (activating a tiny fraction of the store per step), compressed session vectors, memorynative architectures like Recurrent Memory Transformers [Bulatov et al., 2022], and retrieval-free injection via adapters [Modarressi et al., 2024] all point toward cheaper alternatives,”
From Memory for Autonomous LLM Agents:Mechanisms, Evaluation, and Emerging Frontiers (2026) · cites this paper as Bulatov et al., 2022
The different trends in performance as the number of demonstration examples increases are caused by the truncation problem.
“We believe that the different trends are caused by the truncation problem [59], [60].”
From What Makes Good In-Context Demonstrations for Code Intelligence Tasks with LLMs? (2023) · cites this paper as [60]
The Results (p. 6) discuss how performance on copy and reverse tasks changes as the number of sequence segments increases. Our copy does not discuss demonstration examples or attribute differing performance trends to a truncation problem; searches for those terms found no passage addressing that explanation. (LLM, conf=0.76)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
The publisher hasn’t deposited this paper’s reference list.
Research works printed but not linked: 25
- Jimmy Ba, Geoffrey E Hinton, Volodymyr Mnih, Joel Z Leibo, and Catalin Ionescu. Using fast weights to attend to the recent past. In D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 29. Curran Associates, Inc., 2016. URL https://proceedings. neurips.cc/paper/2016/file/9f44e956e3a2b7b5598c625fcc802c36-Paper.pdf .
- Iz Beltagy, Matthew E Peters, and Arman Cohan. Longformer: The long-document transformer. arXiv preprint arXiv:2004.05150 , 2020. · arXiv
- Mikhail S Burtsev, Yuri Kuratov, Anton Peganov, and Grigory V Sapunov. Memory transformer. arXiv preprint arXiv:2006.11527 , 2020. · arXiv
- Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) , pages 4171-4186, 2019. URL https://aclweb.org/anthology/papers/N/N19/ N19-1423/ .
- Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations , 2021. URL https://openreview.net/forum?id=YicbFdNTTy .
- Angela Fan, Thibaut Lavril, Edouard Grave, Armand Joulin, and Sainbayar Sukhbaatar. Addressing some limitations of transformers with feedback memory. arXiv preprint arXiv:2002.09402 , 2020. · arXiv
- Caglar Gulcehre, Sarath Chandar, Kyunghyun Cho, and Yoshua Bengio. Dynamic neural turing machine with soft and hard addressing schemes. arXiv preprint arXiv:1607.00036 , 2016. · arXiv
- Caglar Gulcehre, Sarath Chandar, and Yoshua Bengio. Memory augmented neural networks with wormhole connections. arXiv preprint arXiv:1701.08718 , 2017. · arXiv
- Ankit Gupta and Jonathan Berant. Gmat: Global memory augmentation for transformers. arXiv preprint arXiv:2006.03274 , 2020. · arXiv
- Andrew Jaegle, Sebastian Borgeaud, Jean-Baptiste Alayrac, Carl Doersch, Catalin Ionescu, David Ding, Skanda Koppula, Daniel Zoran, Andrew Brock, Evan Shelhamer, et al. Perceiver io: A general architecture for structured inputs & outputs. arXiv preprint arXiv:2107.14795 , 2021. · arXiv
- Da Ju, Stephen Roller, Sainbayar Sukhbaatar, and Jason Weston. Staircase attention for recurrent processing of sequences. arXiv preprint arXiv:2106.04279 , 2021. · arXiv
- Johannes Kiesel, Maria Mestre, Rishabh Shukla, Emmanuel Vincent, Payam Adineh, David Corney, Benno Stein, and Martin Potthast. Semeval-2019 task 4: Hyperpartisan news detection. In Proceedings of the 13th International Workshop on Semantic Evaluation , pages 829-839, 2019.
- Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR (Poster) , 2015. URL http://arxiv.org/abs/1412.6980 . · arXiv
- Yang Liu, Jiaxiang Liu, Li Chen, Yuxiang Lu, Shikun Feng, Zhida Feng, Yu Sun, Hao Tian, Hua Wu, and Haifeng Wang. Ernie-sparse: Learning hierarchical efficient transformer through regularized self-attention. arXiv preprint arXiv:2203.12276 , 2022. · arXiv
- Pedro Henrique Martins, Zita Marinho, and André FT Martins. ∞ -former: Infinite memory transformer. arXiv preprint arXiv:2109.00301 , 2021. · arXiv
- Warren S McCulloch and Walter Pitts. A logical calculus of the ideas immanent in nervous activity. The bulletin of mathematical biophysics , 5(4):115-133, 1943.
- Yuanliang Meng and Anna Rumshisky. Context-aware neural model for temporal information extraction. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 527-536, 2018.
- Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings . OpenReview.net, 2017. URL https://openreview.net/forum? id=Byj72udxe .
- Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning , volume 139 of Proceedings of Machine Learning Research , pages 8821-8831. PMLR, 18-24 Jul 2021. URL https://proceedings.mlr.press/v139/ ramesh21a.html .
- Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is All you Need. In Advances in neural information processing systems , pages 5998-6008, 2017. URL http://papers.nips.cc/paper/7181-attention-is-all-you-need .
- Paul J Werbos. Backpropagation through time: what it does and how to do it. Proceedings of the IEEE , 78(10): 1550-1560, 1990.
- Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 conference on empirical methods in natural language processing: system demonstrations , pages 38-45, 2020.
- Qingyang Wu, Zhenzhong Lan, Jing Gu, and Zhou Yu. Memformer: The memory-augmented transformer. arXiv preprint arXiv:2010.06891 , 2020. · arXiv
- Peng Xu, Xinchi Chen, Xiaofei Ma, Zhiheng Huang, and Bing Xiang. Contrastive document representation learning with graph attention networks. arXiv preprint arXiv:2110.10778 , 2021. · arXiv
- Manzil Zaheer, Guru Guruganesh, Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, et al. Big bird: Transformers for longer sequences. arXiv preprint arXiv:2007.14062 , 2020. · arXiv
Cited by
Sorted by citations
GENA-LM: a family of open-source foundational DNA language models for long sequences
Fishman, V., Kuratov, Y., Shmelev, A., Petrov, M., Penzar, D., Shepelin, D., Chekanov, N., Kardymon, O. L., & Burtsev, M. (2024) · Nucleic Acids Research · doi:10.1093/nar/gkae1310 · Cited by 142 · see citing works
Self-citationWhat Makes Good In-Context Demonstrations for Code Intelligence Tasks with LLMs?
Gao, S., Wen, X.-C., Gao, C., Wang, W., Zhang, H., & Lyu, M. R. (2023) · 2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase56229.2023.00109 · Cited by 106 · see citing works
Vision Transformers Need Registers
Darcet, T., Oquab, M., Mairal, J., & Bojanowski, P. (2023) · arXiv · doi:10.48550/arxiv.2309.16588 · Cited by 90 · see citing works
“In follow-up work, Bulatov et al. (2022) address complex copy-repeat-reverse tasks.”
Adapting Language Models to Compress Contexts
Chevalier, A., Wettig, A., Ajith, A., & Chen, D. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.232 · Cited by 59 · see citing works
Resource-efficient Algorithms and Systems of Foundation Models: A Survey
Xu, M., Cai, D., Yin, W., Wang, S., Jin, X., & Liu, X. (2024) · ACM Computing Surveys · doi:10.1145/3706418 · Cited by 152 · see citing works
HOOV: Hand Out-Of-View Tracking for Proprioceptive Interaction using Inertial Sensing
Streli, P., Armani, R., Cheng, Y. F., & Holz, C. (2023) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581468 · Cited by 39 · see citing works
Sun, Y., Pang, S., & Zhang, Y. (2024) · Geoenergy Science and Engineering · doi:10.1016/j.geoen.2024.213015 · Cited by 37 · see citing works
Memory-Based Cross-Modal Semantic Alignment Network for Radiology Report Generation
Tao, Y., Ma, L., Yu, J., & Zhang, H. (2024) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2024.3393018 · Cited by 31 · see citing works
Titans: Learning to Memorize at Test Time
Behrouz, A., Zhong, P., & Mirrokni, V. (2025) · arXiv · doi:10.48550/arxiv.2501.00663 · Cited by 24 · see citing works
“…are capable of either memorizing the knowledge abstraction (e.g., persistent memory) [39], or memorizing the data-dependent information (also known as contextual memory), through recurrence [97, 94, 95], Transformers [110, 32–34, 111, 99], gradient [81, 23], or other learning paradigms [112, 113].”
InfluentialTraining Large Language Models to Reason in a Continuous Latent Space
Hao, S., Sukhbaatar, S., Su, D., Li, X., Hu, Z., Weston, J., & Tian, Y. (2024) · arXiv · doi:10.48550/arxiv.2412.06769 · Cited by 18 · see citing works
“…looped transformers (Giannou et al., 2023; Fan et al., 2024) have been proposed to solve algorithmic tasks, which have some similarities to the computing process of continuous thoughts, but we focus on common reasoning tasks and aim at investigating latent reasoning in comparison to language space.”
RWKV: Reinventing RNNs for the Transformer Era
Peng, B., Alcaide, E., Anthony, Q., Albalak, A., Arcadinho, S., Biderman, S., Cao, H., Cheng, X., Chung, M., Grella, M., et al. (2023) · arXiv · doi:10.48550/arxiv.2305.13048 · Cited by 17 · see citing works
“In parallel, RNN-style (Hochreiter and Schmidhuber, 1997; Chung et al., 2014) recursive components have also been modified to increase context length, such as the Recurrent Memory Transformer (Bulatov et al., 2022, 2023) and Linear Recurrent Units (Orvieto et al., 2023).”
MethodState-space modeling in long sequence processing: a survey on recurrence in the transformer era
Tiezzi, M., Casoni, M., Betti, A., Gori, M., & Melacci, S. (2025) · Neural Networks · doi:10.1016/j.neunet.2025.108039 · Cited by 21 · see citing works
Machine learning in computational literary studies
Hatzel, H. O., Stiemer, H., Biemann, C., & Gius, E. (2023) · it - Information Technology · doi:10.1515/itit-2023-0041 · Cited by 21 · see citing works
ChatDB: Augmenting LLMs with Databases as Their Symbolic Memory
Hu, C., Fu, J., Du, C., Luo, S., Zhao, J., & Zhao, H. (2023) · arXiv · doi:10.48550/arxiv.2306.03901 · Cited by 13 · see citing works
“Matrix-based memory employs additional trainable memory tokens (Bulatov et al., 2022, 2023) or trainable memory matrices (Graves et al., 2014).”
BackgroundInfluentialBeyond Attention: Breaking the Limits of Transformer Context Length with Recurrent Memory
Bulatov, A., Kuratov, Y., Kapushev, Y., & Burtsev, M. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i16.29722 · Cited by 12 · see citing works
Self-citationEnhancing Cyberbullying Detection on Social Media Using Transformer Models
Tapaopong, W., Charoenphon, A., Raksasri, J., & Samanchuen, T. (2024) · Technology Innovation Management and Engineering Science International Conference (TIMES-iCON) · doi:10.1109/times-icon61890.2024.10630719 · Cited by 12 · see citing works
Survey on Efficient Large Language Models: Principles, Algorithms, Applications, and Open Issues
Cheng, J., Kang, H., Shao, Y., Li, N., Chen, P., Wang, R., Long, S., Yang, X., & Ma, L. (2025) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2025.3628671 · Cited by 11 · see citing works
Chepurova, A., Bulatov, A., Kuratov, Y., & Burtsev, M. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.352 · Cited by 10 · see citing works
Self-citationGao, C., Chen, X., & Zhang, G. (2025) · Information and Software Technology · doi:10.1016/j.infsof.2025.107803 · Cited by 8 · see citing works
VideoLLaMB: Long Streaming Video Understanding with Recurrent Memory Bridges
Wang, Y., Song, Y., Xie, C., Liu, Y., & Zheng, Z. (2025) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51701.2025.02240 · Cited by 8 · see citing works
In-context Autoencoder for Context Compression in a Large Language Model
Ge, T., Hu, J., Wang, L., Wang, X., Chen, S.-Q., & Wei, F. (2023) · arXiv · doi:10.48550/arxiv.2307.06945 · Cited by 11 · see citing works
“Much previous research (Child et al., 2019; Beltagy et al., 2020; Rae et al., 2019; Choromanski et al., 2020; Bulatov et al., 2022; Zheng et al., 2022; Wu et al., 2022; Bulatov et al., 2023; Ding et al., 2023) attempts to tackle the long context issue through architectural innovations of an LLM.”
BackgroundMem0: Building Production-Ready AI Agents with Scalable Long-Term Memory
Chhikara, P., Khant, D., Aryan, S., Singh, T., & Yadav, D. (2025) · arXiv · doi:10.48550/arxiv.2504.19413 · Cited by 10 · see citing works
Enhancing memory retrieval in generative agents through LLM-trained cross attention networks
Hong, C., & He, Q. (2025) · Frontiers in Psychology · doi:10.3389/fpsyg.2025.1591618 · Cited by 7 · see citing works
DeepPavlov Dream: Platform for Building Generative AI Assistants
Zharikova, D., Kornev, D., Ignatov, F., Talimanchuk, M., Evseev, D., Petukhova, K., Smilga, V., Karpov, D., Shishkina, Y., Kosenko, D., et al. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-demo.58 · Cited by 7 · see citing works
Self-citationBABILong: Testing the Limits of LLMs with Long Context Reasoning-in-a-Haystack
Kuratov, Y., Bulatov, A., Anokhin, P., Rodkin, I., Sorokin, D., Sorokin, A., & Burtsev, M. (2024) · arXiv · doi:10.48550/arxiv.2406.10149 · Cited by 19 · see citing works
“In this work, we augment the Recurrent Memory Transformer (Bulatov et al., 2024) with the ability to retrieve its own past memory tokens.”
BackgroundMethodInfluentialSelf-citationSelfCP: Compressing over-limit prompt via the frozen large language model itself
Gao, J., Cao, Z., & Li, W. (2024) · Information Processing & Management · doi:10.1016/j.ipm.2024.103873 · Cited by 7 · see citing works
Complex QA and language models hybrid architectures, SurveyPreprint
Daull, X., Bellot, P., Bruno, E., Martin, V., & Murisasco, É. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2302.09051 · Cited by 17 · see citing works
Shan, D., Jing, Z., & Luo, Y. (2025) · Applied Soft Computing · doi:10.1016/j.asoc.2025.113448 · Cited by 5 · see citing works
CMT: A Memory Compression Method for Continual Knowledge Learning of Large Language Models
Li, D., Sun, Z., Hu, X., Hu, B., & Zhang, M. (2025) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v39i23.34619 · Cited by 4 · see citing works
Cached Transformers: Improving Transformers with Differentiable Memory Cachde
Zhang, Z., Shao, W., Ge, Y., Wang, X., Gu, J., & Luo, P. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i15.29636 · Cited by 6 · see citing works
Long-Range Language Modeling with Selective Cache
Huang, X., & Hollenstein, N. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.321 · Cited by 3 · see citing works
Recurrent Flash Reinforcement Learning for Dynamic Spectrum Access and Power Control
Lin, K., Yang, H., Yang, T., Pace, P., & Fortino, G. (2024) · IEEE Transactions on Vehicular Technology · doi:10.1109/tvt.2024.3511595 · Cited by 3 · see citing works
Memory-Augmented speech-to-text Translation with Multi-Scale Context Translation Strategy
Yuan, Y., Zhou, Y., & Shi, X. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10447450 · Cited by 3 · see citing works
MIRIX: Multi-Agent Memory System for LLM-Based Agents
Wang, Y., & Chen, X. (2025) · arXiv · doi:10.48550/arxiv.2507.07957 · Cited by 2 · see citing works
“These memory components can reside in various latent forms, including model parameters[30], external memory matrices [8, 21], hidden states [2, 10, 28, 31], soft prompts [3], and key-value caches [38, 9, 16, 37].”
Think before you speak: Training Language Models With Pause Tokens
Goyal, S., Ji, Z., Rawat, A. S., Menon, A. K., Kumar, S., & Nagarajan, V. (2023) · arXiv · doi:10.48550/arxiv.2310.02226 · Cited by 2
“…only as an input has found its use in various forms, most commonly as <cls> (Chang et al., 2023; Liu, 2019; Devlin et al., 2019), <sep> or <mask> in BERT (Devlin et al., 2019) and in a line of work on adding memory to transformers (Burtsev et al., 2020; Bulatov et al., 2022; Darcet et al., 2023).”
BackgroundMVAD: A Multiple Visual Artifact Detector for Video Streaming
Chen, F., Danier, D., Zhang, F., Mackin, A., Collins, A., & Bull, D. (2025) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv61041.2025.00311 · Cited by 2 · see citing works
Space–time recurrent memory network
Nguyen, H. T. T., Kim, C., & Li, F. (2024) · Computer Vision and Image Understanding · doi:10.1016/j.cviu.2024.103943 · Cited by 2 · see citing works
Red Teaming for Multimodal Large Language Models: A SurveyPreprint
Mahato, M., Kumar, A., Singh, K., Kukreja, B., & Nabi, J. (2024) · doi:10.36227/techrxiv.170629758.87975697/v1 · Cited by 2 · see citing works
Uncertainty Guided Global Memory Improves Multi-Hop Question Answering
Sagirova, A., & Burtsev, M. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.262 · Cited by 1 · see citing works
Self-citationWang, X., Xiao, J., Zhang, W., Deng, T., & Wang, Q. (2025) · Expert Systems with Applications · doi:10.1016/j.eswa.2025.129825 · Cited by 1 · see citing works
UAMF: A lightweight and powerful infrared small target detection model
Cui, Y., Li, S., Yang, X., Wang, G., & Wang, Y. (2026) · Applied Soft Computing · doi:10.1016/j.asoc.2026.115121 · Cited by 1 · see citing works
Tian, W., Stockman, S., Zhang, Y., & Werner, M. J. (2026) · Earth s Future · doi:10.1029/2025ef007318 · Cited by 1 · see citing works
Memory for Autonomous LLM Agents:Mechanisms, Evaluation, and Emerging Frontiers
Du, P. (2026) · arXiv · Cited by 1 · see citing works
“Memoriz-ing Transformers [Wu et al., 2022] and Recurrent Memory Transformers [Bulatov et al., 2022] later integrated explicit memory layers directly into the Transformer backbone.”
Adapting Language Models to Compress Contexts
Chevalier, A., Wettig, A., Ajith, A., & Chen, D. (2023) · arXiv · Cited by 59 · see citing works
“We overcome these challenges by adapting the Recurrent Memory Transformer (RMT) (Bulatov et al., 2022), which can be combined with existing language models.”
BackgroundMethodInfluentialSpace Time Recurrent Memory NetworkPreprint
Nguyen, H. S., Kim, C., & Li, F. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4597444
Segmented Recurrent Transformer: An Efficient Sequence-to-Sequence Model
Long, Y., Chowdhury, S., & Roy, K. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.558 · Cited by 1 · see citing works
NarrativeXL: a Large-scale Dataset for Long-Term Memory Models
Moskvichev, A., & Mai, K.-V. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.1005
Šprogar, M. (2023) · Advances in computational intelligence and robotics book series · doi:10.4018/979-8-3693-0230-9.ch005
Memory-Augmenting Decoder-Only Language Models through Encoders (Student Abstract)
Galatolo, A., & Winkle, K. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i21.30444
Sun, Y., Wang, G., Li, W., Kuo, M. M. Y., Bai, Q., & Jiang, J. (2025) · IEEE Congress on Evolutionary Computation (CEC) · doi:10.1109/cec65147.2025.11043068
Document-Level Machine Translation with Context-Aware Memory
Doan, X.-D., Vu, V.-H., Nguyen, N.-D., & Tran, V.-T. (2024) · International Conference on Knowledge and Systems Engineering/Proceedings · doi:10.1109/kse63888.2024.11063610
Wang, Z., Chen, W., Liu, R., Yang, K., Zhou, W., Lin, C., Liu, Q., Sim, M. Y., Zhang, G., Chen, X., et al. (2026) · Synthesis lectures on human language technologies · doi:10.1007/978-3-032-06264-2_13
Xiao, J., Chen, P., Wang, H., Ma, Y., Cheng, H., & Wei, X. H. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.6636965
Lee, S., Kim, H., Kang, S., Kim, H.-J., & Oh, D. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.6636733
A Lightweight and Interpretable Recurrent Vision Transformer for Robust Potato Blight Classification
Sandhya, P., & Venkataramana, B. (2026) · Potato Research · doi:10.1007/s11540-026-10058-3
DreamReg: Belief-Driven World Model for 2D–3D Ultrasound Registration
Kang, L., Zhang, Y., Shan, J., Gong, H., Ding, Q., & Cheng, S. S. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-38186-6_4
Huang, P., Zhang, K., Wang, H., & Wang, J. (2026) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2026.3708967
From Redundancy to Precision: Constructing a Compressed Tree-Based Experiential Case Base for LLMs
Xie, M., Xu, Y., & Li, J. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-33865-5_25
Optimizing AI: A Reference Architecture to Streamline AI Inference and Performance Acceleration
Jovanovic, M., Campbell, M., & Campbell, M. (2026) · Computer · doi:10.1109/mc.2026.3695984
A Survey on Transformer-Based Long-Range Dependency Modeling in Intelligent Document Understanding
Jiang, X., Zhang, Y., & Abbasi, A. (2026) · ACM Computing Surveys · doi:10.1145/3844505
ReRead: Retrieval-Guided Efficient Reading for Long-Context Question Answering
Ling, J., Bao, W., Zhong, Z., Zhou, T., & Wang, J. (2026) · International Conference on Big Data and Information Analytics (BigDIA) · doi:10.1109/bigdia71496.2026.11684612
Chen, G., Zheng, J., Zhao, X., Wang, Z., Li, B., Zhao, J., & 吳政叡 (2026) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2026.116210
Lee, S., Kim, H., Kang, S.-K., Kim, H.-J., & Oh, D. (2026) · Knowledge-Based Systems · doi:10.1016/j.knosys.2026.117032
Zhang, Z., Qiu, Z., Chen, K., Li, H., & Yuki, T. (2026) · Knowledge-Based Systems · doi:10.1016/j.knosys.2026.117188
Token Overflow: Detecting and Mitigating Failures in Soft Context Compression
Belikova, J., Rozhevskii, D., Svirin, D., Makarenko, M., & Panchenko, A. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-41546-2_1
Learning Persistent Referential Identity for Two-Hop Retrieval in Event-Stream Models
Kadyrbek, N., Мансурова, М., Talasbayeva, Z. T., & Zhumasheva, Z. T. (2026) · Machine Learning and Knowledge Extraction · doi:10.3390/make8100334
GENA-LM: A Family of Open-Source Foundational DNA Language Models for Long Sequences
Fishman, V., Kuratov, Y., Shmelev, A., Petrov, M., Penzar, D., Shepelin, D., Chekanov, N., Kardymon, O., & Burtsev, M. (2023) · doi:10.1101/2023.06.12.544594 · Cited by 137 · see citing works
“In the domain of recurrent models, inputs are segmented and sequentially processed, with inter-segment information relayed through prior hidden states [25, 26] or specialized memory [27–29].”
Self-citation
- Authors
- Bulatov, A., Kuratov, Y., & Burtsev, M. S.
- ORCID
- Burtsev, Mikhail S.
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2022
- Pages
- 11079-11091
- ISSN
- 1049-5258 (ISSN-L)
- arXiv
- 2207.06881
- OpenAlex
- W7133228637
- Cited by
- 55
- References
- 0
- Citations by year
- 2026: 17 · 2025: 10 · 2024: 14 · 2023: 14
- FWCI
- 5.25
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- neural information processing systems (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- Analytical Center for the Government of the Russian Federation (000000D730321P5Q0002, 70-2021-00138)
- Moscow Institute of Physics and Technology (70-2021-00138)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Neural Networks and Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Keywords
- memory networks, recurrent neural networks, transformer models, memory enhancement
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
