The DOI matches its DataCite record; 1 of the 1 references it cites were found in a registry or an index.
Retentive Network: A Successor to Transformer for Large Language Models
Open accessAbstract
In this work, we propose Retentive Network (RetNet) as a foundation architecture for large language models, simultaneously achieving training parallelism, low-cost inference, and good performance. We theoretically derive the connection between recurrence and attention. Then we propose the retention mechanism for sequence modeling, which supports three computation paradigms, i.e., parallel, recurrent, and chunkwise recurrent. Specifically, the parallel representation allows for training parallelism. The recurrent representation enables low-cost $O(1)$ inference, which improves decoding throughput, latency, and GPU memory without sacrificing performance. The chunkwise recurrent representation facilitates efficient long-sequence modeling with linear complexity, where each chunk is encoded parallelly while recurrently summarizing the chunks. Experimental results on language modeling show that RetNet achieves favorable scaling results, parallel training, low-cost deployment, and efficient inference. The intriguing properties make RetNet a strong successor to Transformer for large language models. Code will be available at https://aka.ms/retnet.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source doesn’t support the claim as written.
In the citing paper
“RetNet (Sun et al., 2023) and TransNormerLLM (Qin et al., 2024) both utilize a trainable decay factor on the recurrent matrix.”
From MesaNet: Sequence Modeling by Locally Optimal Test-Time Training (2025) · cites this paper as Sun et al., 2023
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“When choosing the kernel as identity matrix (Yutao Sun et al. 2023), the above formulation can also be written in a recurrent format:”
From Titans: Learning to Memorize at Test Time (2025) · cites this paper as Yutao Sun et al. 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“RetNet (Y.Sun et al. 2023) is also based on Linear Attention and very similar to H3, but reduces the inner S4 layer to a special case where the state dimension is 𝑁 = 1.”
From Mamba: Linear-Time Sequence Modeling with Selective State Spaces (2023) · cites this paper as Y.Sun et al. 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Although forget gates have recently significantly improved memory management in these architectures (Peng, Zhang, et al. 2025; Sun, Dong, et al. 2023), their memory's expressivity remains bounded by its linear structure.”
From ATLAS: Learning to Optimally Memorize the Context at Test Time (2025) · cites this paper as Sun, Dong, et al. 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The first generation of modern recurrent architectures (e.g., Linear attention (Katharopoulos et al. 2020), RetNet (Sun et al. 2023), RWKV (Peng et al. 2023), lightening attention (Li et al. 2025)) are based on Hebbian-like learning rules (Hebb 2005).”
From Nested Learning: The Illusion of Deep Learning Architectures (2025) · cites this paper as Sun et al. 2023
Experimental — accuracy not yet measured
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Given different settings for 𝛼 𝑡 (i.e., either is 1, learnable, channel-wise, and/or input-dependent) and also 𝜙 (·) (i.e., identity, polynomial kernels, etc.), the above recurrence recovers different variants of linear attention with Hebbian rule (Katharopoulos et al. 2020; Sun et al. 2023; Arora et al. 2024; Beck et al. 2024; Kacham et al. 2024; Peng et al. 2024).”
From Nested Learning: The Illusion of Deep Learning Architectures (2025) · cites this paper as Sun et al. 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“To this end, in recent years, linear RNNs as alternatives to both Transformers and non-linear RNNs attract much attention mainly due to their parallelizable and linear-time training while maintaining competitive performance (Peng et al. 2025a; Sun et al. 2023; Yang et al. 2024c).”
From It's All Connected: A Journey Through Test-Time Memorization, Attentional Bias, Retention, and Online Optimization (2025) · cites this paper as Sun et al. 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“By absorbing cumulative channel-wise decay into the rank-one erase factors, the recurrence admits a compact WY form with the same high-level chunkwise structure used by efficient delta-rule kernels [14, 15, 16, 17].”
From Gated DeltaNet-2: Decoupling Erase and Write in Linear Attention (2026) · cites this paper as [16]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“We derive a fast-weight update view, a chunkwise WY algorithm with channel-wise decay absorbed into asymmetric erase factors, and a gate-aware backward pass that preserves efficient parallel training.”
From Gated DeltaNet-2: Decoupling Erase and Write in Linear Attention (2026) · cites this paper as [16]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Linear RNNs have recently gained attention as efficient Transformer alternatives due to their parallelizable, linear-time training and comparable performance”
From ATLAS: Learning to Optimally Memorize the Context at Test Time (2025) · cites this paper as Sun, Dong, et al. 2023
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“RetNet (Y.Sun et al. 2023) adds an additional gate to the architecture and uses a simpler SSM, allowing an alternative parallelizable computation path, using a variant of multi-head attention (MHA) instead of convolutions.”
From Mamba: Linear-Time Sequence Modeling with Selective State Spaces (2023) · cites this paper as Y.Sun et al. 2023
The cited source says what the claim attributes to it.
In the citing paper
“Our empirical evaluation is limited to small model sizes, below the threshold of most strong open source LLMs (e.g.Llama ( Touvron et al. 2023)) as well as other recurrent models such as RWKV (B.Peng et al. 2023) and RetNet (Y.Sun et al. 2023), which have been evaluated at the 7B parameter scale and beyond.”
From Mamba: Linear-Time Sequence Modeling with Selective State Spaces (2023) · cites this paper as Y.Sun et al. 2023
The cited source says what the claim attributes to it.
In the citing paper
“Conceptually the closest models to xLSTM are Retention (Sun et al., 2023), RWKV (Peng et al., 2023(Peng et al., , 2024)), and HGRN2 (Qin et al., 2024).”
From xLSTM: Extended Long Short-Term Memory (2024) · cites this paper as Sun et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“Since the dot product between query and normalizer state can be close to zero, we use the absolute value of this dot product and lower bound it by a threshold (typically 1.0) as done previously (Sun et al., 2023).”
From xLSTM: Extended Long Short-Term Memory (2024) · cites this paper as Sun et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“The results show that, in contrast to other methods (Katharopoulos et al., 2020;Sun et al., 2023;Qin et al., 2023;Katsch, 2023;Yang et al., 2023;Qin et al., 2024;Peng et al., 2024), having the gates both learnable and input dependent gives the best results.”
From xLSTM: Extended Long Short-Term Memory (2024) · cites this paper as Sun et al., 2023
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The publisher hasn’t deposited this paper’s reference list.
Research works printed but not linked: 37
- Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arXiv:1607.06450, 2016. · arXiv
- Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sand- hini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In Advances in Neural Information Processing Systems, volume 33, pages 1877-1901.
- Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. Piqa: Reasoning about physical commonsense in natural language. In Thirty-Fourth AAAI Conference on Artificial Intelligence, 2020.
- Mingda Chen, Zewei Chu, Sam Wiseman, and Kevin Gimpel. Summscreen: A dataset for abstractive screenplay summarization. arXiv preprint arXiv:2104.07091, 2021. · arXiv
- Zewen Chi, Li Dong, Shaohan Huang, Damai Dai, Shuming Ma, Barun Patra, Saksham Singhal, Payal Bajaj, Xia Song, Xian-Ling Mao, Heyan Huang, and Furu Wei. On the representation collapse of sparse mixture of experts. In Advances in Neural Information Processing Systems, 2022.
- Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. BoolQ: Exploring the surprising difficulty of natural yes/no questions. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics, pages 2924-2936, 2019.
- Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory-efficient exact attention with io-awareness. Advances in Neural Information Processing Systems, 35:16344-16359, 2022.
- Tri Dao, Daniel Y Fu, Khaled K Saab, Armin W Thomas, Atri Rudra, and Christopher Ré. Hungry hungry hippos: Towards language modeling with state space models. arXiv preprint arXiv:2212.14052, 2022. · arXiv
- Jesse Dodge, Ana Marasović, Gabriel Ilharco, Dirk Groeneveld, Margaret Mitchell, and Matt Gardner. Documenting large webtext corpora: A case study on the colossal clean crawled corpus. In Conference on Empirical Methods in Natural Language Processing, 2021.
- Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. The Pile: An 800GB dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027, 2020. · arXiv
- Albert Gu, Karan Goel, and Christopher Ré. Efficiently modeling long sequences with structured state spaces. arXiv preprint arXiv:2111.00396, 2021. · arXiv
- Luyang Huang, Shuyang Cao, Nikolaus Parulian, Heng Ji, and Lu Wang. Efficient attentions for long document summarization. arXiv preprint arXiv:2104.02112, 2021. · arXiv
- Shaohan Huang, Li Dong, Wenhui Wang, Yaru Hao, Saksham Singhal, Shuming Ma, Tengchao Lv, Lei Cui, Owais Khan Mohammed, Qiang Liu, Kriti Aggarwal, Zewen Chi, Johan Bjorck, Vishrav Chaudhary, Subhojit Som, Xia Song, and Furu Wei. Language is not all you need: Aligning perception with language models. ArXiv, abs/2302.14045, 2023. · arXiv
- Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (GELUs). arXiv: Learning, 2016.
- Yaru Hao, Haoyu Song, Li Dong, Shaohan Huang, Zewen Chi, Wenhui Wang, Shum- ing Ma, and Furu Wei. Language models are general-purpose interfaces. ArXiv, abs/2206.06336, 2022. · arXiv
- Yaru Hao, Yutao Sun, Li Dong, Zhixiong Han, Yuxian Gu, and Furu Wei. Structured prompting: Scaling in-context learning to 1,000 examples. ArXiv, abs/2212.06713, 2022. · arXiv
- Denis Kocetkov, Raymond Li, Loubna Ben Allal, Jia Li, Chenghao Mou, Carlos Muñoz Ferrandis, Yacine Jernite, Margaret Mitchell, Sean Hughes, Thomas Wolf, Dzmitry Bahdanau, Leandro von Werra, and Harm de Vries. The Stack: 3TB of permissively licensed source code. Preprint, 2022.
- Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret. Trans- formers are rnns: Fast autoregressive transformers with linear attention. In International Conference on Machine Learning, pages 5156-5165. PMLR, 2020.
- Hector Levesque, Ernest Davis, and Leora Morgenstern. The winograd schema chal- lenge. In Thirteenth International Conference on the Principles of Knowledge Repre- sentation and Reasoning, 2012.
- Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In Interna- tional Conference on Learning Representations, 2019.
- Nasrin Mostafazadeh, Michael Roth, Annie Louis, Nathanael Chambers, and James Allen. Lsdsem 2017 shared task: The story cloze test. In Proceedings of the 2nd Workshop on Linking Models of Lexical, Sentential and Discourse-level Semantics, pages 46-51, 2017.
- Shuming Ma, Hongyu Wang, Shaohan Huang, Wenhui Wang, Zewen Chi, Li Dong, Alon Benhaim, Barun Patra, Vishrav Chaudhary, Xia Song, and Furu Wei. TorchScale: Transformers at scale. CoRR, abs/2211.13184, 2022. · arXiv
- Antonio Orvieto, Samuel L. Smith, Albert Gu, Anushan Fernando, Caglar Gulcehre, Razvan Pascanu, and Soham De. Resurrecting recurrent neural networks for long sequences. ArXiv, abs/2303.06349, 2023. · arXiv
- Michael Poli, Stefano Massaroli, Eric Nguyen, Daniel Y Fu, Tri Dao, Stephen Baccus, Yoshua Bengio, Stefano Ermon, and Christopher Ré. Hyena hierarchy: Towards larger convolutional language models. arXiv preprint arXiv:2302.10866, 2023. · arXiv
- Zhiliang Peng, Wenhui Wang, Li Dong, Yaru Hao, Shaohan Huang, Shuming Ma, and Furu Wei. Kosmos-2: Grounding multimodal large language models to the world. ArXiv, abs/2306.14824, 2023. · arXiv
- Prajit Ramachandran, Barret Zoph, and Quoc V. Le. Swish: a self-gated activation function. arXiv: Neural and Evolutionary Computing, 2017.
- Yutao Sun, Li Dong, Barun Patra, Shuming Ma, Shaohan Huang, Alon Benhaim, Vishrav Chaudhary, Xia Song, and Furu Wei. A length-extrapolatable transformer. arXiv preprint arXiv:2212.10554, 2022. · arXiv
- Noam M. Shazeer. Fast transformer decoding: One write-head is all you need. ArXiv, abs/1911.02150, 2019. · arXiv
- Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-LM: Training multi-billion parameter language models using model parallelism. arXiv preprint arXiv:1909.08053, 2019. · arXiv
- Uri Shaham, Elad Segal, Maor Ivgi, Avia Efrat, Ori Yoran, Adi Haviv, Ankit Gupta, Wenhan Xiong, Mor Geva, Jonathan Berant, et al. Scrolls: Standardized comparison over long language sequences. arXiv preprint arXiv:2201.03533, 2022. · arXiv
- Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, 4-9 December 2017, Long Beach, CA, USA, pages 6000- 6010, 2017.
- Yuxin Wu and Kaiming He. Group normalization. In Proceedings of the European conference on computer vision (ECCV), pages 3-19, 2018.
- Hongyu Wang, Shuming Ma, Li Dong, Shaohan Huang, Dongdong Zhang, and Furu Wei. DeepNet: Scaling Transformers to 1,000 layers. ArXiv, abs/2203.00555, 2022. · arXiv
- Hongyu Wang, Shuming Ma, Shaohan Huang, Li Dong, Wenhui Wang, Zhiliang Peng, Yu Wu, Payal Bajaj, Saksham Singhal, Alon Benhaim, et al. Foundation transformers. arXiv preprint arXiv:2210.06423, 2022. · arXiv
- Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. SuperGLUE: A stickier benchmark for general-purpose language understanding systems. arXiv preprint arXiv:1905.00537, 2019. · arXiv
- Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 2019.
- Ming Zhong, Da Yin, Tao Yu, Ahmad Zaidi, Mutethia Mutuma, Rahul Jha, Ahmed Has- san Awadallah, Asli Celikyilmaz, Yang Liu, Xipeng Qiu, et al. Qmsum: A new benchmark for query-based multi-domain meeting summarization. arXiv preprint arXiv:2104.05938, 2021. · arXiv
Cited by
Sorted by citations
Items 101–166 of 166
DeepHapNet: a haplotype assembly method based on RetNet and deep spectral clustering
Luo, J., Wang, J., Wei, J., Yan, C., & Luo, H. (2024) · Briefings in Bioinformatics · doi:10.1093/bib/bbae656 · Cited by 2
Self-citationDomínguez-Vidal, J. E., & Sanfeliu, A. (2024) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros58592.2024.10801981 · Cited by 2
Mamba-Hybrid Language Models: Advancing Detection of AI-Generated Text in Academic Contexts
Prajapati, M., Baliarsingh, S. K., Das, S., Dash, A. K., Sarkar, A., & Pradhan, R. (2025) · International Conference on Emerging Systems and Intelligent Computing (ESIC) · doi:10.1109/esic64052.2025.10962699 · Cited by 3
ECoG Dual Context Network for Brain Machine Interfaces
Nagashima, S., Kaneda, K., Wada, Y., Iida, T., Taguchi, M., Hirata, M., & Sugiura, K. (2025) · IEEE Access · doi:10.1109/access.2025.3577407 · Cited by 2
Mamba Models a possible replacement for Transformers?
Das, S., Sen, R., & Devendiran, S. (2024) · Proceedings of the Python in Science Conferences · doi:10.25080/xhdr4700 · Cited by 2
RetViT: Retentive Vision Transformers
Dongre, S., & Mehta, S. (2024) · International Conference on Computing, Communication and Networking Technologies (ICCCNT) · doi:10.1109/icccnt61001.2024.10726145 · Cited by 2
Uncovering Nested Data Parallelism and Data Reuse in DNN Computation with FractalTensor
Liu, S., Qi, C., Cao, Y., Yang, C., Hu, W., Shi, X., Yang, F., & Yang, M. (2024) · ACM SIGOPS Symposium on Operating Systems Principles (SOSP) · doi:10.1145/3694715.3695961 · Cited by 2
USTEP: Spatio-Temporal Predictive Learning Under a Unified View
Tan, C., Wang, J., Gao, Z., Li, S., & Li, S. Z. (2025) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2025.3566420 · Cited by 5
Self-citationZhang, C., Yan, Y. P., Jiang, S., & Zhong, C. (2025) · IEEE Signal Processing Letters · doi:10.1109/lsp.2025.3580496 · Cited by 2
EGLC: Enhancing Global Localization Capability for medical image segmentation
Wan, Y., Zhou, D., & Yan, R. (2025) · Computer Vision and Image Understanding · doi:10.1016/j.cviu.2025.104471 · Cited by 2
Pre-training Retnet of Simulating Entities and Relations as Sentences for Knowledge Graph Reasoning
Cheng, J. S., Meng, T., Ao, X., & Wu, X. (2024) · Asia Conference on Information Engineering (ACIE) · doi:10.1109/acie61839.2024.00008 · Cited by 2
Okada, M., Komatsu, M., & Taniguchi, T. (2024) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros58592.2024.10801976 · Cited by 4
Nested Learning: The Illusion of Deep Learning Architectures
Behrouz, A., Razaviyayn, M., Zhong, P., & Mirrokni, V. (2025) · arXiv · doi:10.48550/arxiv.2512.24695 · Cited by 37
“…keys and values are input tokens (e.g., tokenized text), the choice of objective and the optimization process for solving Equation 1 can result in distinct sequence modeling architectures (see [59] and [58]) such as global/local softmax attention [27], or other modern recurrent models [28, 60, 61].”
InfluentialMamba-3: Improved Sequence Modeling using State Space Principles
Lahoti, A., Li, K. Y., Chen, B., Wang, C., Bick, A., Kolter, J. Z., Dao, T., & Gu, A. (2026) · arXiv · doi:10.48550/arxiv.2603.15569 · Cited by 1
“10 Relative Total State Size”
Long Audio File Speaker Diarization with Feasible End-to-End Models
Huang, K., & Chen, C.-P. (2024) · Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC) · doi:10.1109/apsipaasc63619.2025.10848627 · Cited by 1
Feiyu, M. (2024) · International Computer Conference on Wavelet Active Media Technology and Information Processing (ICCWAMTIP) · doi:10.1109/iccwamtip64812.2024.10873797 · Cited by 1
YoMacs: A high-precision and lightweight algorithm for mouse head-face segmentation
Jin, N., Ye, R., Cai, L., & Chen, Y. (2025) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2025.109940 · Cited by 1
Retention Enhanced Cross-modal Attention for Multi-Hop VQA
Zhu, Z., Ding, F., Chu, C., & Zhong, F. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10889423 · Cited by 1
Zhang, X., Wang, Y., Liu, W., Gu, G., & Li, H. (2024) · IEEE International Conference on High Performance Computing and Communications; IEEE International Conference on Smart City; IEEE International Conference on Data Science and Systems (HPCC/SmartCity/DSS) · doi:10.1109/hpcc64274.2024.00235 · Cited by 1
Wang, R.-Y., Du, J., Niu, S., Yang, G.-B., Gao, T., Pan, J., & Liu, Q. (2025) · Computer Speech & Language · doi:10.1016/j.csl.2025.101863 · Cited by 1
On the Complexity of Formal Reasoning in State Space Models (Extended Abstract)
Alsmann, E., & Lange, M. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-99991-8_11 · Cited by 1
Gao, S., Guo, W., Shi, H., & Peng, R. (2026) · Journal of King Saud University - Computer and Information Sciences · doi:10.1007/s44443-025-00440-5 · Cited by 1
Hyperspectral image classification with retentive network
Paheding, S., Zahan, N., Reyes, A. A., Martinez, E., & Lee, E.-J. (2024) · Pattern Recognition and Tracking · doi:10.1117/12.3013356 · Cited by 1
Variation and evolution analysis of SARS-CoV-2 using self-game sequence optimization
Liu, Z., Shen, Y., Jiang, Y., Zhu, H., Hu, H., Kang, Y., Chen, M., & Li, Z. (2024) · Frontiers in Microbiology · doi:10.3389/fmicb.2024.1485748 · Cited by 1
Optimal RoPE extension via Bayesian Optimization for training-free length generalization
Zhang, X., Hu, S., Zhao, W., Wang, H., Han, X., He, C., Zeng, G., Liu, Z., & Sun, M. (2025) · AI Open · doi:10.1016/j.aiopen.2025.01.002 · Cited by 1
MAR-VO: A Match-and-Refine Framework for UAV’s Monocular Visual Odometry in Planetary Environments
Liu, J., Zhou, B., Wan, X., Pan, Y., Li, Z., & Shao, Y. (2025) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2025.3534581 · Cited by 1
Boost Query-Centric Network Efficiency for Multi-Agent Motion Forecasting
Huang, Y., Chen, K., Tian, W., & Xiong, L. (2025) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2025.3539080 · Cited by 1
SiMBA-TS: Simplified Channel Mixing and Mamba for Long-term Time Series Forecasting
Patro, B. N., & Agneeswaran, V. S. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10890315 · Cited by 1
Yu, X., Wu, Y., Bai, Y., Meng, N., Jin, S., Wu, Q., Chen, L., Wang, N., Song, X., Shen, G., et al. (2025) · CAAI Transactions on Intelligence Technology · doi:10.1049/cit2.70044 · Cited by 1
Surface Defect Detection Algorithm for Printed Circuit Boards Based on SRG-DETRPreprint
Zhou, Z., Lu, Y., & Lv, L. (2024) · Research Square · doi:10.21203/rs.3.rs-4898576/v1 · Cited by 1
Zhang, J., Nolte, N., Sadhukhan, R., Chen, B., & Bottou, L. (2024) · arXiv · doi:10.48550/arxiv.2405.06394 · Cited by 10
“Several recent papers (e.g., Katharopoulos et al., 2020; Peng et al., 2023; Sun et al., 2023; Gu & Dao, 2023) propose transformer alternatives that use efficient recurrences to cut the quadratic computational cost of transformers.”
xLSTM 7B: A Recurrent LLM for Fast and Efficient Inference
Beck, M., Pöppel, K., Lippe, P., Kurle, R., Blies, P. M., Klambauer, G., Böck, S., & Hochreiter, S. (2025) · arXiv · doi:10.48550/arxiv.2503.13427 · Cited by 9
Gated DeltaNet-2: Decoupling Erase and Write in Linear Attention
Hatamizadeh, A., Choi, Y., & Kautz, J. (2026) · arXiv
“Early structured state-space and recurrent models used mostly data-independent transitions [16, 38, 39, 40], while Mamba and Mamba-2 introduced data-dependent selective dynamics and the SSD framework [8, 41].”
Behrouz, A., Razaviyayn, M., Zhong, P., & Mirrokni, V. (2025) · arXiv · doi:10.48550/arxiv.2504.13173 · Cited by 65
MesaNet: Sequence Modeling by Locally Optimal Test-Time Training
von Oswald, J., Scherrer, N., Kobayashi, S., Versari, L., Yang, S., Mittal, S., Schlegel, M., Maile, K., Schimpf, Y., Sieberling, O., et al. (2025) · arXiv
“Since then, numerous works have proposed different designs of the feature map σ ( · ) [54, 20, 88, 76, 97, 25] and key-value normalization [113, 88, 97].”
Gopal, A. M., Pirbadian, A., Carlson, K. D., Lewis, M. A., & Tapson, J. (2026) · arXiv
“PRECOG itself is general: the time-translation-invariance assumption of Theorem 1 holds for any selective-SSM backbone, including Mamba [3], Mamba-2 [4], and related linear-recurrent architectures [5, 6].”
ATLAS: Learning to Optimally Memorize the Context at Test Time
Behrouz, A., Li, Z., Kacham, P., Daliri, M., Deng, Y., Zhong, P., Razaviyayn, M., & Mirrokni, V. (2025) · arXiv · doi:10.48550/arxiv.2505.23735
“…(Katharopoulos et al. 2020) to DeltaNet’s Delta rule (Schlag et al. 2021); (ii) Forget (Retention) Gates, from RetNet’s input-independent gating (Sun, Dong, et al. 2023) to adaptive gating in Titans (Behrouz, Zhong, et al. 2024) and RWKV-7 (Peng, Zhang, et al. 2025); and (iii) Memory…”
FlexPrefill: A Context-Aware Sparse Attention Mechanism for Efficient Long-Sequence Inference
Lai, X., Lu, J., Luo, Y., Ma, Y., & Zhou, X. (2025) · arXiv · doi:10.48550/arxiv.2502.20766
Decision Poisson: From Universal Gravitation to Offline Reinforcement Learning
Cai, H., Zhang, Z., Yao, Z., Mo, K., Chen, D., & Yan, H. (2024) · Lecture notes in computer science · doi:10.1007/978-981-99-9785-5_31
Zhang, H., Xia, D., Qu, J., Li, Y., Shun-shi, Y., Jiang, M., Zhou, L., Tao, X., Fan, X., Zhou, H., et al. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4760256
Attenuation-Based Attention for Improved Novel View Synthesis
Lai, X., Guo, Y., & Liu, Y. (2024) · IEEE International Conference on Electronic Technology, Communication and Information (ICETCI) · doi:10.1109/icetci61221.2024.10594142
Tan, Y., Chen, Z., Liu, Q., Ji, G., Liu, T., & Liu, H. (2024) · International Conference on Computer Vision, Image and Deep Learning (CVIDL) · doi:10.1109/cvidl62147.2024.10603494
Fan, D., & Liu, S. (2024) · Lecture notes in computer science · doi:10.1007/978-981-97-5612-4_31
Edge-Optimized Model for Multimedia Classification using Linguistic Metadata
Bharitkar, S., & Páez, T. (2024) · IEEE International Conference on Acoustics, Speech, and Signal Processing Workshops (ICASSPW) · doi:10.1109/icasspw62465.2024.10626175
From Large Language Models to Generative AI Systems
Tomczak, J. M. (2024) · Springer eBooks · doi:10.1007/978-3-031-64087-2_11
LVS: A Learned Video Storage for Fast and Efficient Video Understanding
Lee, Y., & Park, J. (2024) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw63382.2024.00808
Sequence-to-Sequence Multi-Hop Knowledge Reasoning Based on Retentive Network
Su, R., Zhang, W., & Yu, D. (2024) · International Conference on Computer Information Science and Application Technology (CISAT) · doi:10.1109/cisat62382.2024.10695224
Linear Retention Convolutional Network for Few-Shot Network Threat Detection
Ding, Y., Li, Z., Yang, C., Liang, H., Jia, S., & Li, Z. (2024) · Asia Joint Conference on Information Security (AsiaJCIS) · doi:10.1109/asiajcis64263.2024.00022
Spatio-Temporal Synchronized Retention Networks for Methane Concentration Prediction
Zhang, Z., & Dong, W. (2024) · International Conference on Intelligent Computing and Signal Processing (ICSP) · doi:10.1109/icsp62122.2024.10743332
An Evaluation of Energy-efficient AI Models for Edge Deployment in 12-lead ECG Classification
Baskoro, A., Lin, Y.-C., Nuannimnoi, S., Rijal, H., Traore, G. E. G., Azhar, M. W., & Huang, C. (2024) · IEEE International Conference on Communication, Networks and Satellite (COMNETSAT) · doi:10.1109/comnetsat63286.2024.10862231
Lin, Y., Zhang, Y., Zhang, R., Shao, L., Zheng, X., & Huang, X. (2024) · China Automation Congress (CAC) · doi:10.1109/cac63892.2024.10865276
Takahashi, K., Inamori, R., Ichiji, K., Zhang, Z., Yuwen, Z., & Homma, N. (2025) · SPIE Medical Imaging: Computer-Aided Diagnosis · doi:10.1117/12.3045625
A 77.79 GOPs/W Retentive Network FPGA Inference Accelerator with Optimized Workload
Nian, C., Zhang, W., Farrukh, F. U. D., Niu, L., Jiang, D., Chen, F., & Zhang, C. (2024) · Annual Conference of the IEEE Industrial Electronics Society (IECON) · doi:10.1109/iecon55916.2024.10905965
Radio Frequency‐Retentive Network for Automatic Modulation Classification
Jia, H., Yu, Z., & Yang, J. (2025) · Electronics Letters · doi:10.1049/ell2.70203
Cuffless Blood Pressure Estimation Based on PPG Features
Chen, Y.-X., Sun, M., Lu, C. J., Hung, W.-K., & Lin, Y. (2025) · World Congress on Medical Physics and Biomedical Engineering, September 7 - 12, 2009, Munich, Germany · doi:10.1007/978-3-031-86323-3_11
He, M., Cui, M., Liang, Y., & Liu, H. (2024) · Journal of intelligence and knowledge engineering. · doi:10.62517/jike.202404313
Enhancing Deepfake Detection: A Study Using WavLM and Advanced RawBoost Augmentation Techniques
Tri, N., Hoang, L. N., Viet, P. T., & Trung, K. P. (2024) · RIVF International Conference on Computing and Communication Technologies (RIVF) · doi:10.1109/rivf64335.2024.11009068
Generating Music With Retention Based Latent Transformer Models
Viswanathan, V., Veeravalli, B., & Mohan, G. B. (2025) · International Conference on Communication, Security and Artificial Intelligence (ICCSAI) · doi:10.1109/iccsai64074.2025.11063879
Cai, P., Li, B., Ma, J., Xiao, T., Huo, L., & Jia, X. (2025) · Biomedical Physics & Engineering Express · doi:10.1088/2057-1976/adf3b5
Brain Activity Classification using Retentive Networks with Explainable AI
Hashmi, A. A., Sabharwal, K., Jain, G., & Kumar, N. S. (2026) · Diyala Journal of Engineering Sciences · doi:10.24237/djes.2026.19208
A new time-decay radiomics integrated network (TRINet) for breast cancer risk prediction
Yeoh, H. H., Strand, F., Phan, R. C. W., Rahmat, K., & Tan, M. (2025) · Medical Image Analysis · doi:10.1016/j.media.2025.103829 · Cited by 4
MSFF-YOLO: A Multi-Scale Feature Fusion Network for Aero-Engine Blades Surface Defects Detection
Yuan, J., Zhao, L., Ye, W., Cheng, Y., & Cai, W. (2026) · Journal of Nondestructive Evaluation · doi:10.1007/s10921-025-01321-7
Single-Station Three-Component Seismic Event Localization Using Deep Neural Networks
Guo, M., Li, G., Meng, X., Wang, J., Li, Y., Gu, X., Huang, L., Mo, Z., Peng, H., Peng, K., et al. (2026) · Seismological Research Letters · doi:10.1785/0220250285
Self-citationTransformer Models and their Cross-Domain Applications: A Systematic Review
Pathania, K., Singh, S., & Sharma, A. (2026) · Archives of Computational Methods in Engineering · doi:10.1007/s11831-026-10742-0
Prior-Guided Face Super-Resolution with Weighted Manhattan Spatial Decay Attention
Li, H., Chen, M., & Long, H. (2026) · Arabian Journal for Science and Engineering · doi:10.1007/s13369-026-11749-2
A Survey of Large Language Models
Zhao, W. X., Zhou, K., Li, J., Tang, T., Wang, X., Hou, Y., Min, Y., Zhang, B., Zhang, J., Dong, Z., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.18223 · Cited by 2,090
Metadata
- Authors
- Sun, Y., Dong, L., Huang, S., Ma, S., Xia, Y., Xue, J., Wang, J., & Wei, F.
- Affiliations
- Microsoft (United States) (US)
- Tsinghua University (CN)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2023
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2307.08621
- OpenAlex
- W4384648484
- Cited by
- 142
- References
- 0
- Citations by year
- 2026: 6 · 2025: 61 · 2024: 72 · 2023: 3
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Keywords
- RetNet, large language models, recurrent neural networks, attention, retention mechanism, parallel training, long sequence modeling, linear complexity, decoding throughput, GPU memory efficiency
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
