The DOI matches its DataCite record; 1 of the 1 references it cites were found in a registry or an index.
Retentive Network: A Successor to Transformer for Large Language Models
Open accessAbstract
In this work, we propose Retentive Network (RetNet) as a foundation architecture for large language models, simultaneously achieving training parallelism, low-cost inference, and good performance. We theoretically derive the connection between recurrence and attention. Then we propose the retention mechanism for sequence modeling, which supports three computation paradigms, i.e., parallel, recurrent, and chunkwise recurrent. Specifically, the parallel representation allows for training parallelism. The recurrent representation enables low-cost $O(1)$ inference, which improves decoding throughput, latency, and GPU memory without sacrificing performance. The chunkwise recurrent representation facilitates efficient long-sequence modeling with linear complexity, where each chunk is encoded parallelly while recurrently summarizing the chunks. Experimental results on language modeling show that RetNet achieves favorable scaling results, parallel training, low-cost deployment, and efficient inference. The intriguing properties make RetNet a strong successor to Transformer for large language models. Code will be available at https://aka.ms/retnet.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source doesn’t support the claim as written.
In the citing paper
“RetNet (Sun et al., 2023) and TransNormerLLM (Qin et al., 2024) both utilize a trainable decay factor on the recurrent matrix.”
From MesaNet: Sequence Modeling by Locally Optimal Test-Time Training (2025) · cites this paper as Sun et al., 2023
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“When choosing the kernel as identity matrix (Yutao Sun et al. 2023), the above formulation can also be written in a recurrent format:”
From Titans: Learning to Memorize at Test Time (2025) · cites this paper as Yutao Sun et al. 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“RetNet (Y.Sun et al. 2023) is also based on Linear Attention and very similar to H3, but reduces the inner S4 layer to a special case where the state dimension is 𝑁 = 1.”
From Mamba: Linear-Time Sequence Modeling with Selective State Spaces (2023) · cites this paper as Y.Sun et al. 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Although forget gates have recently significantly improved memory management in these architectures (Peng, Zhang, et al. 2025; Sun, Dong, et al. 2023), their memory's expressivity remains bounded by its linear structure.”
From ATLAS: Learning to Optimally Memorize the Context at Test Time (2025) · cites this paper as Sun, Dong, et al. 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The first generation of modern recurrent architectures (e.g., Linear attention (Katharopoulos et al. 2020), RetNet (Sun et al. 2023), RWKV (Peng et al. 2023), lightening attention (Li et al. 2025)) are based on Hebbian-like learning rules (Hebb 2005).”
From Nested Learning: The Illusion of Deep Learning Architectures (2025) · cites this paper as Sun et al. 2023
Experimental — accuracy not yet measured
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Given different settings for 𝛼 𝑡 (i.e., either is 1, learnable, channel-wise, and/or input-dependent) and also 𝜙 (·) (i.e., identity, polynomial kernels, etc.), the above recurrence recovers different variants of linear attention with Hebbian rule (Katharopoulos et al. 2020; Sun et al. 2023; Arora et al. 2024; Beck et al. 2024; Kacham et al. 2024; Peng et al. 2024).”
From Nested Learning: The Illusion of Deep Learning Architectures (2025) · cites this paper as Sun et al. 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“To this end, in recent years, linear RNNs as alternatives to both Transformers and non-linear RNNs attract much attention mainly due to their parallelizable and linear-time training while maintaining competitive performance (Peng et al. 2025a; Sun et al. 2023; Yang et al. 2024c).”
From It's All Connected: A Journey Through Test-Time Memorization, Attentional Bias, Retention, and Online Optimization (2025) · cites this paper as Sun et al. 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“By absorbing cumulative channel-wise decay into the rank-one erase factors, the recurrence admits a compact WY form with the same high-level chunkwise structure used by efficient delta-rule kernels [14, 15, 16, 17].”
From Gated DeltaNet-2: Decoupling Erase and Write in Linear Attention (2026) · cites this paper as [16]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“We derive a fast-weight update view, a chunkwise WY algorithm with channel-wise decay absorbed into asymmetric erase factors, and a gate-aware backward pass that preserves efficient parallel training.”
From Gated DeltaNet-2: Decoupling Erase and Write in Linear Attention (2026) · cites this paper as [16]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Linear RNNs have recently gained attention as efficient Transformer alternatives due to their parallelizable, linear-time training and comparable performance”
From ATLAS: Learning to Optimally Memorize the Context at Test Time (2025) · cites this paper as Sun, Dong, et al. 2023
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“RetNet (Y.Sun et al. 2023) adds an additional gate to the architecture and uses a simpler SSM, allowing an alternative parallelizable computation path, using a variant of multi-head attention (MHA) instead of convolutions.”
From Mamba: Linear-Time Sequence Modeling with Selective State Spaces (2023) · cites this paper as Y.Sun et al. 2023
The cited source says what the claim attributes to it.
In the citing paper
“Our empirical evaluation is limited to small model sizes, below the threshold of most strong open source LLMs (e.g.Llama ( Touvron et al. 2023)) as well as other recurrent models such as RWKV (B.Peng et al. 2023) and RetNet (Y.Sun et al. 2023), which have been evaluated at the 7B parameter scale and beyond.”
From Mamba: Linear-Time Sequence Modeling with Selective State Spaces (2023) · cites this paper as Y.Sun et al. 2023
The cited source says what the claim attributes to it.
In the citing paper
“Conceptually the closest models to xLSTM are Retention (Sun et al., 2023), RWKV (Peng et al., 2023(Peng et al., , 2024)), and HGRN2 (Qin et al., 2024).”
From xLSTM: Extended Long Short-Term Memory (2024) · cites this paper as Sun et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“Since the dot product between query and normalizer state can be close to zero, we use the absolute value of this dot product and lower bound it by a threshold (typically 1.0) as done previously (Sun et al., 2023).”
From xLSTM: Extended Long Short-Term Memory (2024) · cites this paper as Sun et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“The results show that, in contrast to other methods (Katharopoulos et al., 2020;Sun et al., 2023;Qin et al., 2023;Katsch, 2023;Yang et al., 2023;Qin et al., 2024;Peng et al., 2024), having the gates both learnable and input dependent gives the best results.”
From xLSTM: Extended Long Short-Term Memory (2024) · cites this paper as Sun et al., 2023
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The publisher hasn’t deposited this paper’s reference list.
Research works printed but not linked: 37
- Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arXiv:1607.06450, 2016. · arXiv
- Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sand- hini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In Advances in Neural Information Processing Systems, volume 33, pages 1877-1901.
- Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. Piqa: Reasoning about physical commonsense in natural language. In Thirty-Fourth AAAI Conference on Artificial Intelligence, 2020.
- Mingda Chen, Zewei Chu, Sam Wiseman, and Kevin Gimpel. Summscreen: A dataset for abstractive screenplay summarization. arXiv preprint arXiv:2104.07091, 2021. · arXiv
- Zewen Chi, Li Dong, Shaohan Huang, Damai Dai, Shuming Ma, Barun Patra, Saksham Singhal, Payal Bajaj, Xia Song, Xian-Ling Mao, Heyan Huang, and Furu Wei. On the representation collapse of sparse mixture of experts. In Advances in Neural Information Processing Systems, 2022.
- Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. BoolQ: Exploring the surprising difficulty of natural yes/no questions. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics, pages 2924-2936, 2019.
- Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory-efficient exact attention with io-awareness. Advances in Neural Information Processing Systems, 35:16344-16359, 2022.
- Tri Dao, Daniel Y Fu, Khaled K Saab, Armin W Thomas, Atri Rudra, and Christopher Ré. Hungry hungry hippos: Towards language modeling with state space models. arXiv preprint arXiv:2212.14052, 2022. · arXiv
- Jesse Dodge, Ana Marasović, Gabriel Ilharco, Dirk Groeneveld, Margaret Mitchell, and Matt Gardner. Documenting large webtext corpora: A case study on the colossal clean crawled corpus. In Conference on Empirical Methods in Natural Language Processing, 2021.
- Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. The Pile: An 800GB dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027, 2020. · arXiv
- Albert Gu, Karan Goel, and Christopher Ré. Efficiently modeling long sequences with structured state spaces. arXiv preprint arXiv:2111.00396, 2021. · arXiv
- Luyang Huang, Shuyang Cao, Nikolaus Parulian, Heng Ji, and Lu Wang. Efficient attentions for long document summarization. arXiv preprint arXiv:2104.02112, 2021. · arXiv
- Shaohan Huang, Li Dong, Wenhui Wang, Yaru Hao, Saksham Singhal, Shuming Ma, Tengchao Lv, Lei Cui, Owais Khan Mohammed, Qiang Liu, Kriti Aggarwal, Zewen Chi, Johan Bjorck, Vishrav Chaudhary, Subhojit Som, Xia Song, and Furu Wei. Language is not all you need: Aligning perception with language models. ArXiv, abs/2302.14045, 2023. · arXiv
- Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (GELUs). arXiv: Learning, 2016.
- Yaru Hao, Haoyu Song, Li Dong, Shaohan Huang, Zewen Chi, Wenhui Wang, Shum- ing Ma, and Furu Wei. Language models are general-purpose interfaces. ArXiv, abs/2206.06336, 2022. · arXiv
- Yaru Hao, Yutao Sun, Li Dong, Zhixiong Han, Yuxian Gu, and Furu Wei. Structured prompting: Scaling in-context learning to 1,000 examples. ArXiv, abs/2212.06713, 2022. · arXiv
- Denis Kocetkov, Raymond Li, Loubna Ben Allal, Jia Li, Chenghao Mou, Carlos Muñoz Ferrandis, Yacine Jernite, Margaret Mitchell, Sean Hughes, Thomas Wolf, Dzmitry Bahdanau, Leandro von Werra, and Harm de Vries. The Stack: 3TB of permissively licensed source code. Preprint, 2022.
- Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret. Trans- formers are rnns: Fast autoregressive transformers with linear attention. In International Conference on Machine Learning, pages 5156-5165. PMLR, 2020.
- Hector Levesque, Ernest Davis, and Leora Morgenstern. The winograd schema chal- lenge. In Thirteenth International Conference on the Principles of Knowledge Repre- sentation and Reasoning, 2012.
- Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In Interna- tional Conference on Learning Representations, 2019.
- Nasrin Mostafazadeh, Michael Roth, Annie Louis, Nathanael Chambers, and James Allen. Lsdsem 2017 shared task: The story cloze test. In Proceedings of the 2nd Workshop on Linking Models of Lexical, Sentential and Discourse-level Semantics, pages 46-51, 2017.
- Shuming Ma, Hongyu Wang, Shaohan Huang, Wenhui Wang, Zewen Chi, Li Dong, Alon Benhaim, Barun Patra, Vishrav Chaudhary, Xia Song, and Furu Wei. TorchScale: Transformers at scale. CoRR, abs/2211.13184, 2022. · arXiv
- Antonio Orvieto, Samuel L. Smith, Albert Gu, Anushan Fernando, Caglar Gulcehre, Razvan Pascanu, and Soham De. Resurrecting recurrent neural networks for long sequences. ArXiv, abs/2303.06349, 2023. · arXiv
- Michael Poli, Stefano Massaroli, Eric Nguyen, Daniel Y Fu, Tri Dao, Stephen Baccus, Yoshua Bengio, Stefano Ermon, and Christopher Ré. Hyena hierarchy: Towards larger convolutional language models. arXiv preprint arXiv:2302.10866, 2023. · arXiv
- Zhiliang Peng, Wenhui Wang, Li Dong, Yaru Hao, Shaohan Huang, Shuming Ma, and Furu Wei. Kosmos-2: Grounding multimodal large language models to the world. ArXiv, abs/2306.14824, 2023. · arXiv
- Prajit Ramachandran, Barret Zoph, and Quoc V. Le. Swish: a self-gated activation function. arXiv: Neural and Evolutionary Computing, 2017.
- Yutao Sun, Li Dong, Barun Patra, Shuming Ma, Shaohan Huang, Alon Benhaim, Vishrav Chaudhary, Xia Song, and Furu Wei. A length-extrapolatable transformer. arXiv preprint arXiv:2212.10554, 2022. · arXiv
- Noam M. Shazeer. Fast transformer decoding: One write-head is all you need. ArXiv, abs/1911.02150, 2019. · arXiv
- Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-LM: Training multi-billion parameter language models using model parallelism. arXiv preprint arXiv:1909.08053, 2019. · arXiv
- Uri Shaham, Elad Segal, Maor Ivgi, Avia Efrat, Ori Yoran, Adi Haviv, Ankit Gupta, Wenhan Xiong, Mor Geva, Jonathan Berant, et al. Scrolls: Standardized comparison over long language sequences. arXiv preprint arXiv:2201.03533, 2022. · arXiv
- Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, 4-9 December 2017, Long Beach, CA, USA, pages 6000- 6010, 2017.
- Yuxin Wu and Kaiming He. Group normalization. In Proceedings of the European conference on computer vision (ECCV), pages 3-19, 2018.
- Hongyu Wang, Shuming Ma, Li Dong, Shaohan Huang, Dongdong Zhang, and Furu Wei. DeepNet: Scaling Transformers to 1,000 layers. ArXiv, abs/2203.00555, 2022. · arXiv
- Hongyu Wang, Shuming Ma, Shaohan Huang, Li Dong, Wenhui Wang, Zhiliang Peng, Yu Wu, Payal Bajaj, Saksham Singhal, Alon Benhaim, et al. Foundation transformers. arXiv preprint arXiv:2210.06423, 2022. · arXiv
- Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. SuperGLUE: A stickier benchmark for general-purpose language understanding systems. arXiv preprint arXiv:1905.00537, 2019. · arXiv
- Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 2019.
- Ming Zhong, Da Yin, Tao Yu, Ahmad Zaidi, Mutethia Mutuma, Rahul Jha, Ahmed Has- san Awadallah, Asli Celikyilmaz, Yang Liu, Xipeng Qiu, et al. Qmsum: A new benchmark for query-based multi-domain meeting summarization. arXiv preprint arXiv:2104.05938, 2021. · arXiv
Cited by
Sorted by citations
A Survey of Large Language Models
Zhao, W. X., Zhou, K., Li, J., Tang, T., Dong, Z., Hou, Y., Zhang, B., Min, Y., Zhang, J., Liu, P., et al. (2026) · Frontiers of Computer Science · doi:10.1007/s11704-026-60308-3 · Cited by 2,093
Mamba: Linear-Time Sequence Modeling with Selective State Spaces
Gu, A., & Dao, T. (2023) · arXiv · doi:10.48550/arxiv.2312.00752 · Cited by 1,579
VideoMamba: State Space Model for Efficient Video Understanding
Li, K., Li, X., Wang, Y., He, Y., Wang, Y., Wang, L., & Qiao, Y. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-73347-5_14 · Cited by 500
Foundation Models Defining a New Era in Vision: A Survey and Outlook
Awais, M., Naseer, M., Khan, S., Anwer, R. M., Cholakkal, H., Shah, M., Yang, M.-H., & Khan, F. S. (2025) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3506283 · Cited by 304
xLSTM: Extended Long Short-Term Memory
Beck, M., Pöppel, K., Spanring, M., Auer, A., Prudnikova, O., Kopp, M., Klambauer, G., Brandstetter, J., & Hochreiter, S. (2024) · Advances in Neural Information Processing Systems 37 · doi:10.52202/079017-3417 · Cited by 286
RMT: Retentive Networks Meet Vision Transformers
Fan, Q., Huang, H., Chen, M., Liu, H., & He, R. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.00539 · Cited by 296
From Matching to Generation: A Survey on Generative Information Retrieval
Li, X., Jin, J., Zhou, Y., Zhang, Y., Zhang, P., Zhu, Y., & Dou, Z. (2025) · ACM Transactions on Information Systems · doi:10.1145/3722552 · Cited by 172
LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compression
Jiang, H., Wu, Q., Luo, X., Li, D., Lin, C.-Y., Yang, Y., & Qiu, L. (2024) · Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2024.acl-long.91 · Cited by 400
Transformers and genome language models
Consens, M. E., Dufault, C., Wainberg, M., Forster, D., Karimzadeh, M., Goodarzi, H., Theis, F. J., Moses, A. M., & Wang, B. (2025) · Nature Machine Intelligence · doi:10.1038/s42256-025-01007-9 · Cited by 170
xLSTM: Extended Long Short-Term Memory
Beck, M., Pöppel, K., Spanring, M., Auer, A., Prudnikova, O., Kopp, M., Klambauer, G., Brandstetter, J., & Hochreiter, S. (2024) · arXiv · doi:10.48550/arxiv.2405.04517 · Cited by 299
CellFM: a large-scale foundation model pre-trained on transcriptomics of 100 million human cells
Zeng, Y., Xie, J., Shangguan, N., Wei, Z., Li, W., Su, Y., Yang, S., Zhang, C., Zhang, J., Fang, N., et al. (2025) · Nature Communications · doi:10.1038/s41467-025-59926-5 · Cited by 100
MultiSenseSeg: A Cost-Effective Unified Multimodal Semantic Segmentation Model for Remote Sensing
Wang, Q., Chen, W., Huang, Z., Tang, H., & Yang, L. (2024) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2024.3390750 · Cited by 58
Peng, L., Liu, X., Chen, M., Liao, W., Mao, J., & Zhou, L. (2024) · Journal of Chemical Information and Modeling · doi:10.1021/acs.jcim.4c00957 · Cited by 53
Novel applications of Convolutional Neural Networks in the age of Transformers
Ersavas, T., Smith, M. A., & Mattick, J. S. (2024) · Scientific Reports · doi:10.1038/s41598-024-60709-z · Cited by 64
Ma, S., Zhang, Y., Peng, L., Sun, C., Ding, B., & Zhu, Y. (2025) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2025.3560928 · Cited by 54
Self-citationWang, R., Gao, Z., Zhang, L., Yue, S., & Gao, Z. (2025) · Computer Science Review · doi:10.1016/j.cosrev.2025.100755 · Cited by 61
Resource-efficient Algorithms and Systems of Foundation Models: A Survey
Xu, M., Cai, D., Yin, W., Wang, S., Jin, X., & Liu, X. (2024) · ACM Computing Surveys · doi:10.1145/3706418 · Cited by 255
Knowledge-Empowered, Collaborative, and Co-Evolving AI Models: The Post-LLM Roadmap
Wu, F., Shen, T., Bäck, T., Chen, J., Huang, G., Jin, Y., Kuang, K., Li, M., Lu, C., Miao, J., et al. (2024) · Engineering · doi:10.1016/j.eng.2024.12.008 · Cited by 47
Multichannel Long-Term Streaming Neural Speech Enhancement for Static and Moving Speakers
Quan, C., & Li, X. (2024) · IEEE Signal Processing Letters · doi:10.1109/lsp.2024.3418714 · Cited by 55
TransRAD: Retentive Vision Transformer for Enhanced Radar Object Detection
Cheng, L., & Cao, S. (2025) · IEEE Transactions on Radar Systems · doi:10.1109/trs.2025.3537604 · Cited by 40
Hadamard Product in Deep Learning: Introduction, Advances and Challenges
Chrysos, G. G., Wu, Y., Pascanu, R., Torr, P. H. S., & Cevher, V. (2025) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2025.3560423 · Cited by 43
An outline of Prognostics and health management Large Model: Concepts, Paradigms, and challenges
Tao, L., Li, S., Liu, H., Huang, Q., Ma, L., Ning, G., Chen, Y., Wu, Y., Li, B., Zhang, W., et al. (2025) · Mechanical Systems and Signal Processing · doi:10.1016/j.ymssp.2025.112683 · Cited by 37
Multimodal fusion for large-scale traffic prediction with heterogeneous retentive networks
Yan, Y., Cui, S., Liu, J., Zhao, Y., Zhou, B., & Kuo, Y. (2024) · Information Fusion · doi:10.1016/j.inffus.2024.102695 · Cited by 33
BFA-YOLO: A balanced multiscale object detection network for building façade elements detection
Chen, Y., Wang, T., Chen, G., Zhu, K., Tan, X., Wang, J., Guo, W., Wang, Q., Luo, X., & Zhang, X. (2025) · Advanced Engineering Informatics · doi:10.1016/j.aei.2025.103289 · Cited by 34
Self-citationFourier-Deformable Convolution Network for Road Segmentation From Remote Sensing Images
Liu, H., Zhou, X., Wang, C., Chen, S., & Kong, H. (2024) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2024.3476087 · Cited by 28
Xu, Z., Chen, N., Li, G., Li, J., Zhu, H., & Zhu, Z. (2024) · Knowledge-Based Systems · doi:10.1016/j.knosys.2024.112826 · Cited by 21
Span-level bidirectional retention scheme for aspect sentiment triplet extraction
Yang, X., Peng, T., Bi, H., & Han, J. (2024) · Information Processing & Management · doi:10.1016/j.ipm.2024.103823 · Cited by 21
Luo, X., Liu, D., Kong, H., Huai, S., Chen, H., Xiong, G., & Liu, W. (2024) · ACM Transactions on Embedded Computing Systems · doi:10.1145/3701728 · Cited by 27
Toward a deeper understanding: RetNet viewed through Convolution
Li, C., & Zhang, C. (2024) · Pattern Recognition · doi:10.1016/j.patcog.2024.110625 · Cited by 20
Titans: Learning to Memorize at Test Time
Behrouz, A., Zhong, P., & Mirrokni, V. (2025) · arXiv · doi:10.48550/arxiv.2501.00663 · Cited by 24
“More specifically in language tasks, we compare with Transformer++ [48], RetNet [49], Gated Linear Attention (GLA) [9], Mamba [45], Mamba2 [30], DeltaNet [38], TTT [21], and Gated DeltaNet [35].”
Zhu, X., Sun, S., Lin, L., Wu, Y., & Ma, X. (2024) · Reviews in the Neurosciences · doi:10.1515/revneuro-2024-0088 · Cited by 18
Retentive network-based time series anomaly detection in cyber-physical systems
Min, Z., Xiao, Q., Abbas, M., & Zhang, D. (2025) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2025.110215 · Cited by 17
Jiang, B., Liao, M., Yun, Z., Li, G., Cheng, S., Wang, X., & Xia, Q. (2025) · Image and Vision Computing · doi:10.1016/j.imavis.2025.105463 · Cited by 20
Chen, G., Yang, F., Peng, W., Fan, Y., & Lyu, X. (2024) · Applied Energy · doi:10.1016/j.apenergy.2024.124266 · Cited by 15
Tan, M. J. T., Lichlyter, D. A., Maravilla, N. M. A. T., Schrock, W. J., Ting, F. I. L., Choa-Go, J. M., Francisco, K. K. Y., Byers, M. C., Karim, H. B. A., & AlDahoul, N. (2025) · Frontiers in Digital Health · doi:10.3389/fdgth.2025.1535018 · Cited by 19
The evolution of language models: From N-Grams to LLMs, and beyond
Raeini, M. (2025) · Natural Language Processing Journal · doi:10.1016/j.nlp.2025.100168 · Cited by 26
MV-YOLO: An Efficient Small Object Detection Framework Based on Mamba
Wu, S., Lu, X.-Y., Guo, C., & Guo, H. (2025) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2025.3584955 · Cited by 27
RULER: What's the Real Context Size of Your Long-Context Language Models?
Hsieh, C.-P., Sun, S., Kriman, S., Acharya, S., Rekesh, D., Jia, F., Zhang, Y., & Ginsburg, B. (2024) · arXiv · Cited by 1,068
“Finally, novel architectures (Gu et al., 2022; Fu et al., 2023a; Poli et al., 2023; Fu et al., 2023b; Sun et al., 2023a; Beck et al., 2024; Sun et al., 2024) such as Mamba (Gu & Dao, 2023) and RWKV (Peng et al., 2023) have also been proposed to efficiently handle long-context input.”
MethodFIGLUT: An Energy-Efficient Accelerator Design for FP-INT GEMM Using Look-Up Tables
Park, G., Kwon, H., Kim, J., Bae, J., Park, B., Lee, D., & Lee, Y. (2025) · IEEE International Symposium on High-Performance Computer Architecture (HPCA) · doi:10.1109/hpca61900.2025.00085 · Cited by 16
Graph Reconstruction Attention Fusion Network for Multimodal Sentiment Analysis
Hu, R., Yi, J., Chen, L., & Jin, Z. (2024) · IEEE Transactions on Industrial Informatics · doi:10.1109/tii.2024.3452204 · Cited by 13
Yuan, S., Wang, G., Mu, B., & Zhou, F. (2024) · Advances in Atmospheric Sciences · doi:10.1007/s00376-024-3313-9 · Cited by 14
LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compression
Jiang, H., Wu, Q., Luo, X., Li, D., Lin, C.-Y., Yang, Y., & Qiu, L. (2023) · arXiv · Cited by 12
“Add the subsequence x i,j = { x i , x i +1 , ..., x j } to the response y rec .”
BackgroundYou, Z., Zhang, P., Zheng, J. Y., & Cartlidge, J. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10447394 · Cited by 12
Beyond Attention: Breaking the Limits of Transformer Context Length with Recurrent Memory
Bulatov, A., Kuratov, Y., Kapushev, Y., & Burtsev, M. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i16.29722 · Cited by 12
GR-pKa: a message-passing neural network with retention mechanism for pKa prediction
Miao, R., Liu, D., Mao, L., Chen, X., Zhang, L., Yuan, Z., Shi, S., Li, H., & Li, S. (2024) · Briefings in Bioinformatics · doi:10.1093/bib/bbae408 · Cited by 12
Intelligent coal gangue identification: A novel amplitude frequency sensitive neural network
Zhang, Z., Zhu, Z., Meng, B., Yang, Z., Wu, M., Cheng, X., Li, B., & Liu, H. (2025) · Expert Systems with Applications · doi:10.1016/j.eswa.2025.126880 · Cited by 12
Sparks of Large Audio Models: A Survey and Outlook
Latif, S., Shoukat, M., Shamshad, F., Usama, M., Ren, Y., Cuayáhuitl, H., Wang, W., Zhang, X., Togneri, R., Cambria, E., et al. (2023) · arXiv · doi:10.48550/arxiv.2308.12792 · Cited by 17
“These techniques are efficient attention mechanisms [277– 280], length generalisation [178, 281–284], and transformer alternatives [285–288].”
BackgroundLarge language models for PHM: a review of optimization techniques and applications
Yu, T., Tang, J., Yu, Q., Li, L., Liu, Y., & Poler, R. (2025) · Autonomous Intelligent Systems · doi:10.1007/s43684-025-00100-5 · Cited by 13
Zhou, Z., Lu, Y., & Lv, L. (2025) · Annals of the New York Academy of Sciences · doi:10.1111/nyas.15332 · Cited by 11
AI for enhanced water quality data imputation: a deep learning perspective
Banjara, I. P., Poudel, S., Pariyar, K., Upreti, D., Zafeirakou, A., & Paudel, S. R. (2025) · Environmental Science Advances · doi:10.1039/d4va00367e · Cited by 10
Neishabouri, A., Bauer, J., Abdollahi, A., Debus, J., & Mairani, A. (2025) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2025.109777 · Cited by 10
HumMUSS: Human Motion Understanding Using State Space Models
Mondal, A. K., Alletto, S., & Tomè, D. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.00225 · Cited by 9
Transformers significantly improve splice site prediction
Jónsson, B. A., Halldorsson, G. H., Árdal, S., Rögnvaldsson, S., Einarsson, E., Sulem, P., Gudbjartsson, D. F., Melsted, P., Stefánsson, K., & Ulfarsson, M. O. (2024) · Communications Biology · doi:10.1038/s42003-024-07298-9 · Cited by 13
GRetNet: Gaussian Retentive Network for Hyperspectral Image Classification
Han, Z., Xu, S., Gao, L., Li, Z., & Zhang, B. (2024) · IEEE Geoscience and Remote Sensing Letters · doi:10.1109/lgrs.2024.3450536 · Cited by 8
Huang, Y., Zhao, W., Fu, Y., Zhu, L., & Yu, L. (2024) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2024.3475587 · Cited by 10
RetSeg3D: Retention-based 3D semantic segmentation for autonomous driving
Erabati, G. K., & Araújo, H. J. (2024) · Computer Vision and Image Understanding · doi:10.1016/j.cviu.2024.104231 · Cited by 8
Cho, S., Kim, J., & Kim, J. H. (2024) · IEEE Access · doi:10.1109/access.2024.3502219 · Cited by 13
HFA-Net: hierarchical feature aggregation network for micro-expression recognition
Zhang, M., Yang, W., Wang, L., Wu, Z., & Chen, D. (2025) · Complex & Intelligent Systems · doi:10.1007/s40747-025-01804-0 · Cited by 9
Back to recurrent processing at the crossroad of transformers and state-space models
Tiezzi, M., Casoni, M., Betti, A., Guidi, T., Gori, M., & Melacci, S. (2025) · Nature Machine Intelligence · doi:10.1038/s42256-025-01034-6 · Cited by 33
DouN-GNN:Double nodes graph neural network for few-shot learning
Zhang, Y., Zhou, X., Wang, N., Tang, J. H., & Tao, X. (2024) · Neurocomputing · doi:10.1016/j.neucom.2024.128970 · Cited by 7
Zhou, W., Kamata, S., Wang, H., Lu, P., & Zhang, M. (2024) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2024.3384403 · Cited by 7
RAMPGrasp: Retentive Attention-Based Multiscale Perception Grasp Detection Network
Huang, J., Liu, X., Zhu, Q. X., Wang, Y., Feng, M., Zhou, J., Zhou, Z., Chen, L., & Wang, D. (2025) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2025.3548814 · Cited by 7
L-Eval: Instituting Standardized Evaluation for Long Context Language Models
An, C., Gong, S., Zhong, M., Zhao, X., Li, M., Zhang, J., Kong, L., & Qiu, X. (2023) · arXiv · doi:10.48550/arxiv.2307.11088 · Cited by 6
LRetUNet: A U-Net-based retentive network for single-channel speech enhancement
Zhang, Y., Zhang, Z., Guo, W., Chen, W., Liu, Z., & Liu, H. (2025) · Computer Speech & Language · doi:10.1016/j.csl.2025.101798 · Cited by 6
Identifying different classes of geoacoustic events using machine learning
Wang, J., Zuo, Y., Dong, L., & Yan, X. (2025) · International Journal of Rock Mechanics and Mining Sciences · doi:10.1016/j.ijrmms.2025.106144 · Cited by 7
Self-citationA multiscale process-aware retention network for fault prediction in mixed-model production
Chen, M., Yu, R., Liang, Z., Li, K., & Qi, H. R. (2025) · Computers in Industry · doi:10.1016/j.compind.2025.104313 · Cited by 6
Li, Y., Liu, X., Ferreira, V. G., Balzter, H., Zhou, H., Ge, Y., Lai, M., Chu, S., Han, D., & Gu, Z. (2025) · International Journal of Applied Earth Observation and Geoinformation · doi:10.1016/j.jag.2025.104666 · Cited by 6
How Big Can It Get? A comparative analysis of LLMs in architecture and scaling
Yousri, R., & Safwat, S. (2023) · International Conference on Computer and Applications (ICCA) · doi:10.1109/icca59364.2023.10401818 · Cited by 6
Hyperspectral image classification using hybrid convolutional-based cross-patch retentive network
Arya, R. K., Peddi, R., & Srivastava, R. (2025) · Computer Vision and Image Understanding · doi:10.1016/j.cviu.2025.104382 · Cited by 6
Li, Y., Lai, L., Shangguan, Y., Iandola, F., Ni, Z., Chang, E., Shi, Y., & Chandra, V. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10447827 · Cited by 6
Lessons from the Trenches on Reproducible Evaluation of Language Models
Biderman, S., Schoelkopf, H., Sutawika, L., Gao, L., Tow, J., Abbasi, B., Aji, A. F., Ammanamanchi, P. S., Black, S., Clive, J., et al. (2024) · arXiv · doi:10.48550/arxiv.2405.14782 · Cited by 9
Yu, S., Yu, C., Wang, F., Zhang, B., & Tian, H. (2025) · Measurement Science and Technology · doi:10.1088/1361-6501/adf2c6 · Cited by 6
Huang, L., Zhang, Z., Yu, Y., & Tang, B. (2025) · IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing · doi:10.1109/jstars.2025.3586948 · Cited by 5
Yoon, D., Kim, T., Lee, J. W., & Rhu, M. (2024) · IEEE Computer Architecture Letters · doi:10.1109/lca.2024.3422492 · Cited by 5
Guo, C., Li, X., Ma, J., Li, Y., Liu, Y., Qi, H., Zhang, L., & Jin, Y. (2024) · IEEE Transactions on Fuzzy Systems · doi:10.1109/tfuzz.2024.3425664 · Cited by 5
PFPRNet: A Phase-Wise Feature Pyramid With Retention Network for Polyp Segmentation
Chu, J., Liu, W., Tian, Q., & Lu, W. D. (2024) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2024.3500026 · Cited by 5
Weird-Net: Weighted Relative Distance Attention for Efficient and Robust Sequence Processing
Hu, L., Zhan, F., Huang, W., Gan, W., Hu, H., He, H., & Han, K. (2024) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2024.3513912 · Cited by 5
Liquid white box model as an explainable AI for surgery
Riva-Cambrin, H. A., Singh, R., Lama, S., & Sutherland, G. R. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01769-x · Cited by 6
Tan, Y., Liu, Q., Liu, T., Liu, H., Wang, S.-M., & Chen, Z. (2024) · Applied Sciences · doi:10.3390/app14104259 · Cited by 5
Annealing Temporal–Spatial Contrastive Learning for multi-view Online Action Detection
Tan, Y., Xie, L., Jing, S., Fang, S., & Zhang, K. (2024) · Knowledge-Based Systems · doi:10.1016/j.knosys.2024.112523 · Cited by 4
Toward a dynamic tree-Mamba encoder for UAV tracking with vision-language
Du, G., Zhou, P., Yadikar, N., Aysa, A., & Ubul, K. (2025) · Knowledge-Based Systems · doi:10.1016/j.knosys.2025.113731 · Cited by 5
Spatial-Temporal Retentive Heterogeneous Graph Convolutional Network for Traffic Flow Prediction
Wang, Z., Long, B., & Xiao, J. (2024) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn60899.2024.10651032 · Cited by 4
AIST AIRC Submissions to the WMT23 Shared Task
Rikters, M., & Miwa, M. (2023) · Conference on Machine Translation (WMT) · doi:10.18653/v1/2023.wmt-1.13 · Cited by 4
A Comparative Review of Advanced Techniques for Financial Sentiment Analysis
Xiao, W. (2024) · International Conference on Modeling, Natural Language Processing and Machine Learning (CMNM) · doi:10.1145/3677779.3677791 · Cited by 4
ST-RetNet: A Long-Term Spatial-Temporal Traffic Flow Prediction Method
Long, B., Zhu, W., & Xiao, J. (2024) · Lecture notes in computer science · doi:10.1007/978-981-97-8502-5_1 · Cited by 5
Advances in edge-cloud collaboration and evolution for large-small models
Wang, Y., Shen, T., Zhang, S., Wu, F., Zhao, Z., Cai, H., Lyu, C., Ma, L., Yang, C., & Wu, F. L. (2024) · Journal of Image and Graphics · doi:10.11834/jig.240011 · Cited by 8
Improved YOLOv8 Method for Multi-scale Pothole Detection
Chang, J., Chen, Z., & Xia, E. (2024) · Lecture notes in computer science · doi:10.1007/978-981-97-5612-4_33 · Cited by 4
Toward a Deeper Understanding: Retnet Viewed Through ConvolutionPreprint
Li, C., & Zhang, C. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4637493 · Cited by 4
Dong, Z., Tang, T., Li, J., Zhao, W. X., & Wen, J.-R. (2023) · arXiv · doi:10.48550/arxiv.2309.13345 · Cited by 7
“S4 (Gu et al., 2022), DSS (Gupta et al., 2022), and GSS (Mehta et al., 2023) employed State Space Models, while RetNet (Sun et al., 2023) utilized retention mechanism, achieving parallel training and recurrent inference for long sequences.”
MethodConvolutional Retentive Network for EEG Decoding
Wang, J., Hang, W., Liang, S., Wang, Q., Chen, B., & Qin, J. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10890343 · Cited by 3
Self-citationXu, J., Hong, H., Zhou, Y., Hu, X., & Duan, S. (2025) · Neurocomputing · doi:10.1016/j.neucom.2025.130517 · Cited by 3
Khosravi, F., & Shadaram, M. (2025) · IEEE Access · doi:10.1109/access.2025.3585076 · Cited by 3
LS-EEND: Long-Form Streaming End-to-End Neural Diarization With Online Attractor Extraction
Liang, D., & Li, X. (2025) · IEEE Transactions on Audio Speech and Language Processing · doi:10.1109/taslpro.2025.3597446 · Cited by 3
Dong, Y., Wu, T., & Song, C. (2024) · Journal of Computing and Information Science in Engineering · doi:10.1115/1.4067524 · Cited by 3
Wei, H. L. C. L. Y. H. Y., Han, F., Liu, Q., & Han, H. (2025) · Complex System Modeling and Simulation · doi:10.23919/csms.2024.0029 · Cited by 3
Li, Z., Xia, T., Chang, Y., & Wu, Y. (2025) · Neurocomputing · doi:10.1016/j.neucom.2025.130711 · Cited by 5
Adaptive gated meta graph retention network: A model for urban traffic flow prediction
Li, X., & Bao, Y. (2025) · Expert Systems with Applications · doi:10.1016/j.eswa.2025.129703 · Cited by 3
Vertical slice equal sampling and transformer network for point cloud ground filtering
Wen, W., Yang, R., Tan, J., Liu, H., & Tan, J. (2024) · International Journal of Remote Sensing · doi:10.1080/01431161.2024.2368927 · Cited by 3
Retentive neural quantum states: efficient ansätze for ab initio quantum chemistry
Knitter, O., Zhao, D., STOKES, J. M., Ganahl, M., Leichenauer, S., & Veerapaneni, S. (2025) · Machine Learning Science and Technology · doi:10.1088/2632-2153/adcb88 · Cited by 3
Keep the Cost Down: A Review on Methods to Optimize LLM' s KV-Cache Consumption
Shi, L., Zhang, H., Yao, Y., Li, Z., & Zhao, H. (2024) · arXiv · doi:10.48550/arxiv.2407.18003 · Cited by 2
Metadata
- Authors
- Sun, Y., Dong, L., Huang, S., Ma, S., Xia, Y., Xue, J., Wang, J., & Wei, F.
- Affiliations
- Microsoft (United States) (US)
- Tsinghua University (CN)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2023
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2307.08621
- OpenAlex
- W4384648484
- Cited by
- 142
- References
- 0
- Citations by year
- 2026: 6 · 2025: 61 · 2024: 72 · 2023: 3
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Keywords
- RetNet, large language models, recurrent neural networks, attention, retention mechanism, parallel training, long sequence modeling, linear complexity, decoding throughput, GPU memory efficiency
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
