Retentive Network: A Successor to Transformer for Large Language Models
Open accessAbstract
In this work, we propose Retentive Network (RetNet) as a foundation architecture for large language models, simultaneously achieving training parallelism, low-cost inference, and good performance. We theoretically derive the connection between recurrence and attention. Then we propose the retention mechanism for sequence modeling, which supports three computation paradigms, i.e., parallel, recurrent, and chunkwise recurrent. Specifically, the parallel representation allows for training parallelism. The recurrent representation enables low-cost $O(1)$ inference, which improves decoding throughput, latency, and GPU memory without sacrificing performance. The chunkwise recurrent representation facilitates efficient long-sequence modeling with linear complexity, where each chunk is encoded parallelly while recurrently summarizing the chunks. Experimental results on language modeling show that RetNet achieves favorable scaling results, parallel training, low-cost deployment, and efficient inference. The intriguing properties make RetNet a strong successor to Transformer for large language models. Code will be available at https://aka.ms/retnet.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
RetNet (Sun et al., 2023) and TransNormerLLM (Qin et al., 2024) both utilize a trainable decay factor on the recurrent matrix.
“RetNet (Sun et al., 2023) and TransNormerLLM (Qin et al., 2024) both utilize a trainable decay factor on the recurrent matrix.”
From MesaNet: Sequence Modeling by Locally Optimal Test-Time Training (2025) · cites this paper as Sun et al., 2023
Section 2.2, “Gated Multi-Scale Retention” (page not specified), says RetNet assigns different decay factors γ to its heads but keeps them fixed. The recurrent-retention pseudocode shows decay applied to the past state; it is the claim that this factor is trainable that conflicts with the source. This verdict concerns RetNet, not the separately cited TransNormerLLM. (LLM, conf=0.95)
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredWhen choosing the kernel as identity matrix, the linear attention formulation can be written in a recurrent format.
“When choosing the kernel as identity matrix (Yutao Sun et al. 2023), the above formulation can also be written in a recurrent format:”
From Titans: Learning to Memorize at Test Time (2025) · cites this paper as Yutao Sun et al. 2023
The “Transformer RetNet” section says linearized attention uses kernels and can be rewritten in recurrent form. Section 2.1 also gives parallel and recurrent forms of the paper’s retention mechanism. Our copy does not specify choosing an identity matrix as the kernel, so that condition is unsupported. (LLM, conf=0.82)
Experimental — accuracy not yet measuredRetNet is based on Linear Attention and very similar to H3, but reduces the inner S4 layer to a special case where the state dimension is N=1.
“RetNet (Y.Sun et al. 2023) is also based on Linear Attention and very similar to H3, but reduces the inner S4 layer to a special case where the state dimension is 𝑁 = 1.”
From Mamba: Linear-Time Sequence Modeling with Selective State Spaces (2023) · cites this paper as Y.Sun et al. 2023
Section 2.1 describes RetNet’s retention mechanism using query–key products and recurrent states, and the “S4” subsection relates its formulation to S4. But the paper does not say RetNet reduces an inner S4 layer to state dimension N=1; Section 3.5 compares RetNet with H3 without claiming they are architecturally very similar. The “Linear Attention” subsection also distinguishes retention from approaches that approximate softmax. (LLM, conf=0.83)
Experimental — accuracy not yet measuredAlthough forget gates have improved memory management in linear RNNs, their memory's expressivity remains bounded by its linear structure.
“Although forget gates have recently significantly improved memory management in these architectures (Peng, Zhang, et al. 2025; Sun, Dong, et al. 2023), their memory's expressivity remains bounded by its linear structure.”
From ATLAS: Learning to Optimally Memorize the Context at Test Time (2025) · cites this paper as Sun, Dong, et al. 2023
Section 3.6 (page unavailable) reports that RetNet’s gating module improves model capability and that decay improves language-modeling performance. Section 2.1 (page unavailable) describes a linear recurrent state update. The paper does not establish that forget gates improve memory management in linear RNNs or that their memory’s expressivity remains bounded by its linear structure. (LLM, conf=0.74)
Experimental — accuracy not yet measuredThe first generation of modern recurrent architectures are based on Hebbian-like learning rules.
“The first generation of modern recurrent architectures (e.g., Linear attention (Katharopoulos et al. 2020), RetNet (Sun et al. 2023), RWKV (Peng et al. 2023), lightening attention (Li et al. 2025)) are based on Hebbian-like learning rules (Hebb 2005).”
From Nested Learning: The Illusion of Deep Learning Architectures (2025) · cites this paper as Sun et al. 2023
Section 2.1 (page unavailable) describes RetNet as recurrent and gives a state update that adds an outer product of key and value vectors, consistent with a Hebbian-like update. The paper does not describe this as a Hebbian learning rule or classify RetNet as part of a “first generation” of modern recurrent architectures. (LLM, conf=0.78)
Experimental — accuracy not yet measured
Different settings for α_t and φ(·) recover different variants of linear attention with Hebbian rule.
“Given different settings for 𝛼 𝑡 (i.e., either is 1, learnable, channel-wise, and/or input-dependent) and also 𝜙 (·) (i.e., identity, polynomial kernels, etc.), the above recurrence recovers different variants of linear attention with Hebbian rule (Katharopoulos et al. 2020; Sun et al. 2023; Arora et al. 2024; Beck et al. 2024; Kacham et al. 2024; Peng et al. 2024).”
From Nested Learning: The Illusion of Deep Learning Architectures (2025) · cites this paper as Sun et al. 2023
Section 2.1, Eq. (6) (page not supplied), gives a recurrence of the stated form: a decay factor multiplies the previous state, and a key–value product updates it. Section 2.2 specifies fixed, head-dependent decay rates. The paper does not present its recurrence as recovering the full range of αₜ and φ(·) settings listed in the citing sentence, or describe the update as a Hebbian rule. (LLM, conf=0.84)
Experimental — accuracy not yet measuredLinear RNNs attract attention mainly due to their parallelizable and linear-time training while maintaining competitive performance.
“To this end, in recent years, linear RNNs as alternatives to both Transformers and non-linear RNNs attract much attention mainly due to their parallelizable and linear-time training while maintaining competitive performance (Peng et al. 2025a; Sun et al. 2023; Yang et al. 2024c).”
From It's All Connected: A Journey Through Test-Time Memorization, Attentional Bias, Retention, and Online Optimization (2025) · cites this paper as Sun et al. 2023
Section 2.1 describes RetNet’s recurrent and parallel forms, and Section 3.5 gives linear-in-sequence-length complexity for its chunkwise training while reporting efficient training without sacrificing performance. These passages support the claim for RetNet, but do not establish that linear RNNs generally have all three properties or that these are the main reason they attract attention. Page numbers were not supplied. (LLM, conf=0.85)
Experimental — accuracy not yet measuredBy absorbing cumulative channel-wise decay into the rank-one erase factors, the recurrence admits a compact WY form with the same high-level chunkwise structure used by efficient delta-rule kernels.
“By absorbing cumulative channel-wise decay into the rank-one erase factors, the recurrence admits a compact WY form with the same high-level chunkwise structure used by efficient delta-rule kernels [14, 15, 16, 17].”
From Gated DeltaNet-2: Decoupling Erase and Write in Linear Attention (2026) · cites this paper as [16]
Section 2.1 (page not supplied) describes a decayed recurrence and a chunkwise computation that combines parallel processing within chunks with recurrent state passed between chunks. It does not describe absorbing channel-wise decay into rank-one erase factors, a compact WY form, or delta-rule kernels. The source supports the high-level chunkwise analogy, not the stated derivation. (LLM, conf=0.84)
Experimental — accuracy not yet measuredThe paper derives a fast-weight update view, a chunkwise WY algorithm with channel-wise decay absorbed into asymmetric erase factors, and a gate-aware backward pass that preserves efficient parallel training.
“We derive a fast-weight update view, a chunkwise WY algorithm with channel-wise decay absorbed into asymmetric erase factors, and a gate-aware backward pass that preserves efficient parallel training.”
From Gated DeltaNet-2: Decoupling Erase and Write in Linear Attention (2026) · cites this paper as [16]
Section 2.1 (Retention; page not provided) gives a recurrent key–value state update and a chunkwise formulation that enables parallel training, supporting the fast-weight-like and chunkwise aspects. Section 2.2 uses a swish gate and head-specific decay rates, but our copy does not describe a WY algorithm, asymmetric erase factors, or a gate-aware backward pass. (LLM, conf=0.88)
Experimental — accuracy not yet measuredLinear RNNs have parallelizable, linear-time training and comparable performance.
“Linear RNNs have recently gained attention as efficient Transformer alternatives due to their parallelizable, linear-time training and comparable performance”
From ATLAS: Learning to Optimally Memorize the Context at Test Time (2025) · cites this paper as Sun, Dong, et al. 2023
The Transformer RetNet and 3.5 Comparison with Transformer Variants sections support parallel training, Transformer-comparable performance, and training complexity linear in sequence length for RetNet specifically. The claim extends those properties to linear RNNs generally; Table 1 in Training Parallelization instead marks recurrent neural networks as lacking training parallelization and strong performance. Page numbers are unavailable in our copy. (LLM, conf=0.87)
Experimental — accuracy not yet measuredRetNet adds an additional gate to the architecture and uses a simpler SSM, allowing an alternative parallelizable computation path, using a variant of multi-head attention instead of convolutions.
“RetNet (Y.Sun et al. 2023) adds an additional gate to the architecture and uses a simpler SSM, allowing an alternative parallelizable computation path, using a variant of multi-head attention (MHA) instead of convolutions.”
From Mamba: Linear-Time Sequence Modeling with Selective State Spaces (2023) · cites this paper as Y.Sun et al. 2023
Our empirical evaluation is limited to small model sizes, below the threshold of most strong open source LLMs as well as other recurrent models such as RWKV and RetNet, which have been evaluated at the 7B parameter scale and beyond.
“Our empirical evaluation is limited to small model sizes, below the threshold of most strong open source LLMs (e.g.Llama ( Touvron et al. 2023)) as well as other recurrent models such as RWKV (B.Peng et al. 2023) and RetNet (Y.Sun et al. 2023), which have been evaluated at the 7B parameter scale and beyond.”
From Mamba: Linear-Time Sequence Modeling with Selective State Spaces (2023) · cites this paper as Y.Sun et al. 2023
Conceptually the closest models to xLSTM are Retention, RWKV, and HGRN2.
“Conceptually the closest models to xLSTM are Retention (Sun et al., 2023), RWKV (Peng et al., 2023(Peng et al., , 2024)), and HGRN2 (Qin et al., 2024).”
From xLSTM: Extended Long Short-Term Memory (2024) · cites this paper as Sun et al., 2023
Since the dot product between query and normalizer state can be close to zero, the absolute value of this dot product is used and lower bounded by a threshold (typically 1.0).
“Since the dot product between query and normalizer state can be close to zero, we use the absolute value of this dot product and lower bound it by a threshold (typically 1.0) as done previously (Sun et al., 2023).”
From xLSTM: Extended Long Short-Term Memory (2024) · cites this paper as Sun et al., 2023
Having the gates both learnable and input dependent gives the best results for mLSTM, in contrast to other methods.
“The results show that, in contrast to other methods (Katharopoulos et al., 2020;Sun et al., 2023;Qin et al., 2023;Katsch, 2023;Yang et al., 2023;Qin et al., 2024;Peng et al., 2024), having the gates both learnable and input dependent gives the best results.”
From xLSTM: Extended Long Short-Term Memory (2024) · cites this paper as Sun et al., 2023
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Still being checked.
Research works printed but not linked: 38
- Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arXiv:1607.06450, 2016. · arXiv
- Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sand- hini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In Advances in Neural Information Processing Systems, volume 33, pages 1877-1901.
- Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. Piqa: Reasoning about physical commonsense in natural language. In Thirty-Fourth AAAI Conference on Artificial Intelligence, 2020.
- Mingda Chen, Zewei Chu, Sam Wiseman, and Kevin Gimpel. Summscreen: A dataset for abstractive screenplay summarization. arXiv preprint arXiv:2104.07091, 2021. · arXiv
- Zewen Chi, Li Dong, Shaohan Huang, Damai Dai, Shuming Ma, Barun Patra, Saksham Singhal, Payal Bajaj, Xia Song, Xian-Ling Mao, Heyan Huang, and Furu Wei. On the representation collapse of sparse mixture of experts. In Advances in Neural Information Processing Systems, 2022.
- Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. BoolQ: Exploring the surprising difficulty of natural yes/no questions. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics, pages 2924-2936, 2019.
- Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory-efficient exact attention with io-awareness. Advances in Neural Information Processing Systems, 35:16344-16359, 2022.
- Tri Dao, Daniel Y Fu, Khaled K Saab, Armin W Thomas, Atri Rudra, and Christopher Ré. Hungry hungry hippos: Towards language modeling with state space models. arXiv preprint arXiv:2212.14052, 2022. · arXiv
- Jesse Dodge, Ana Marasović, Gabriel Ilharco, Dirk Groeneveld, Margaret Mitchell, and Matt Gardner. Documenting large webtext corpora: A case study on the colossal clean crawled corpus. In Conference on Empirical Methods in Natural Language Processing, 2021.
- Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. The Pile: An 800GB dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027, 2020. · arXiv
- Albert Gu, Karan Goel, and Christopher Ré. Efficiently modeling long sequences with structured state spaces. arXiv preprint arXiv:2111.00396, 2021. · arXiv
- Luyang Huang, Shuyang Cao, Nikolaus Parulian, Heng Ji, and Lu Wang. Efficient attentions for long document summarization. arXiv preprint arXiv:2104.02112, 2021. · arXiv
- Shaohan Huang, Li Dong, Wenhui Wang, Yaru Hao, Saksham Singhal, Shuming Ma, Tengchao Lv, Lei Cui, Owais Khan Mohammed, Qiang Liu, Kriti Aggarwal, Zewen Chi, Johan Bjorck, Vishrav Chaudhary, Subhojit Som, Xia Song, and Furu Wei. Language is not all you need: Aligning perception with language models. ArXiv, abs/2302.14045, 2023. · arXiv
- Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (GELUs). arXiv: Learning, 2016.
- Yaru Hao, Haoyu Song, Li Dong, Shaohan Huang, Zewen Chi, Wenhui Wang, Shum- ing Ma, and Furu Wei. Language models are general-purpose interfaces. ArXiv, abs/2206.06336, 2022. · arXiv
- Yaru Hao, Yutao Sun, Li Dong, Zhixiong Han, Yuxian Gu, and Furu Wei. Structured prompting: Scaling in-context learning to 1,000 examples. ArXiv, abs/2212.06713, 2022. · arXiv
- Denis Kocetkov, Raymond Li, Loubna Ben Allal, Jia Li, Chenghao Mou, Carlos Muñoz Ferrandis, Yacine Jernite, Margaret Mitchell, Sean Hughes, Thomas Wolf, Dzmitry Bahdanau, Leandro von Werra, and Harm de Vries. The Stack: 3TB of permissively licensed source code. Preprint, 2022.
- Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret. Trans- formers are rnns: Fast autoregressive transformers with linear attention. In International Conference on Machine Learning, pages 5156-5165. PMLR, 2020.
- Hector Levesque, Ernest Davis, and Leora Morgenstern. The winograd schema chal- lenge. In Thirteenth International Conference on the Principles of Knowledge Repre- sentation and Reasoning, 2012.
- Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In Interna- tional Conference on Learning Representations, 2019.
- Nasrin Mostafazadeh, Michael Roth, Annie Louis, Nathanael Chambers, and James Allen. Lsdsem 2017 shared task: The story cloze test. In Proceedings of the 2nd Workshop on Linking Models of Lexical, Sentential and Discourse-level Semantics, pages 46-51, 2017.
- Shuming Ma, Hongyu Wang, Shaohan Huang, Wenhui Wang, Zewen Chi, Li Dong, Alon Benhaim, Barun Patra, Vishrav Chaudhary, Xia Song, and Furu Wei. TorchScale: Transformers at scale. CoRR, abs/2211.13184, 2022. · arXiv
- Antonio Orvieto, Samuel L. Smith, Albert Gu, Anushan Fernando, Caglar Gulcehre, Razvan Pascanu, and Soham De. Resurrecting recurrent neural networks for long sequences. ArXiv, abs/2303.06349, 2023. · arXiv
- Michael Poli, Stefano Massaroli, Eric Nguyen, Daniel Y Fu, Tri Dao, Stephen Baccus, Yoshua Bengio, Stefano Ermon, and Christopher Ré. Hyena hierarchy: Towards larger convolutional language models. arXiv preprint arXiv:2302.10866, 2023. · arXiv
- Zhiliang Peng, Wenhui Wang, Li Dong, Yaru Hao, Shaohan Huang, Shuming Ma, and Furu Wei. Kosmos-2: Grounding multimodal large language models to the world. ArXiv, abs/2306.14824, 2023. · arXiv
- Prajit Ramachandran, Barret Zoph, and Quoc V. Le. Swish: a self-gated activation function. arXiv: Neural and Evolutionary Computing, 2017.
- Yutao Sun, Li Dong, Barun Patra, Shuming Ma, Shaohan Huang, Alon Benhaim, Vishrav Chaudhary, Xia Song, and Furu Wei. A length-extrapolatable transformer. arXiv preprint arXiv:2212.10554, 2022. · arXiv
- Noam M. Shazeer. Fast transformer decoding: One write-head is all you need. ArXiv, abs/1911.02150, 2019. · arXiv
- Jianlin Su, Yu Lu, Shengfeng Pan, Bo Wen, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. arXiv preprint arXiv:2104.09864, 2021. · arXiv
- Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-LM: Training multi-billion parameter language models using model parallelism. arXiv preprint arXiv:1909.08053, 2019. · arXiv
- Uri Shaham, Elad Segal, Maor Ivgi, Avia Efrat, Ori Yoran, Adi Haviv, Ankit Gupta, Wenhan Xiong, Mor Geva, Jonathan Berant, et al. Scrolls: Standardized comparison over long language sequences. arXiv preprint arXiv:2201.03533, 2022. · arXiv
- Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, 4-9 December 2017, Long Beach, CA, USA, pages 6000- 6010, 2017.
- Yuxin Wu and Kaiming He. Group normalization. In Proceedings of the European conference on computer vision (ECCV), pages 3-19, 2018.
- Hongyu Wang, Shuming Ma, Li Dong, Shaohan Huang, Dongdong Zhang, and Furu Wei. DeepNet: Scaling Transformers to 1,000 layers. ArXiv, abs/2203.00555, 2022. · arXiv
- Hongyu Wang, Shuming Ma, Shaohan Huang, Li Dong, Wenhui Wang, Zhiliang Peng, Yu Wu, Payal Bajaj, Saksham Singhal, Alon Benhaim, et al. Foundation transformers. arXiv preprint arXiv:2210.06423, 2022. · arXiv
- Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. SuperGLUE: A stickier benchmark for general-purpose language understanding systems. arXiv preprint arXiv:1905.00537, 2019. · arXiv
- Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 2019.
- Ming Zhong, Da Yin, Tao Yu, Ahmad Zaidi, Mutethia Mutuma, Rahul Jha, Ahmed Has- san Awadallah, Asli Celikyilmaz, Yang Liu, Xipeng Qiu, et al. Qmsum: A new benchmark for query-based multi-domain meeting summarization. arXiv preprint arXiv:2104.05938, 2021. · arXiv
Cited by
Sorted by citations
A Survey of Large Language Models
Zhao, W. X., Zhou, K., Li, J., Tang, T., Dong, Z., Hou, Y., Zhang, B., Min, Y., Zhang, J., Liu, P., et al. (2026) · Frontiers of Computer Science · doi:10.1007/s11704-026-60308-3 · Cited by 2,093 · see citing works
Mamba: Linear-Time Sequence Modeling with Selective State Spaces
Gu, A., & Dao, T. (2023) · arXiv · doi:10.48550/arxiv.2312.00752 · Cited by 1,579 · see citing works
VideoMamba: State Space Model for Efficient Video Understanding
Li, K., Li, X., Wang, Y., He, Y., Wang, Y., Wang, L., & Qiao, Y. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-73347-5_14 · Cited by 378 · see citing works
Foundation Models Defining a New Era in Vision: A Survey and Outlook
Awais, M., Naseer, M., Khan, S., Anwer, R. M., Cholakkal, H., Shah, M., Yang, M.-H., & Khan, F. S. (2025) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3506283 · Cited by 304 · see citing works
xLSTM: Extended Long Short-Term Memory
Beck, M., Pöppel, K., Spanring, M., Auer, A., Prudnikova, O., Kopp, M., Klambauer, G., Brandstetter, J., & Hochreiter, S. (2024) · Advances in Neural Information Processing Systems 37 · doi:10.52202/079017-3417 · Cited by 286 · see citing works
RMT: Retentive Networks Meet Vision Transformers
Fan, Q., Huang, H., Chen, M., Liu, H., & He, R. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.00539 · Cited by 278 · see citing works
LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compression
Jiang, H., Wu, Q., Luo, X., Li, D., Lin, C.-Y., Yang, Y., & Qiu, L. (2024) · Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2024.acl-long.91 · Cited by 96 · see citing works
Transformers and genome language models
Consens, M. E., Dufault, C., Wainberg, M., Forster, D., Karimzadeh, M., Goodarzi, H., Theis, F. J., Moses, A. M., & Wang, B. (2025) · Nature Machine Intelligence · doi:10.1038/s42256-025-01007-9 · Cited by 148 · see citing works
xLSTM: Extended Long Short-Term Memory
Beck, M., Pöppel, K., Spanring, M., Auer, A., Prudnikova, O., Kopp, M., Klambauer, G., Brandstetter, J., & Hochreiter, S. (2024) · arXiv · doi:10.48550/arxiv.2405.04517 · Cited by 299 · see citing works
CellFM: a large-scale foundation model pre-trained on transcriptomics of 100 million human cells
Zeng, Y., Xie, J., Shangguan, N., Wei, Z., Li, W., Su, Y., Yang, S., Zhang, C., Zhang, J., Fang, N., et al. (2025) · Nature Communications · doi:10.1038/s41467-025-59926-5 · Cited by 93 · see citing works
From Matching to Generation: A Survey on Generative Information Retrieval
Li, X., Jin, J., Zhou, Y., Zhang, Y., Zhang, P., Zhu, Y., & Dou, Z. (2025) · ACM Transactions on Information Systems · doi:10.1145/3722552 · Cited by 124 · see citing works
MultiSenseSeg: A Cost-Effective Unified Multimodal Semantic Segmentation Model for Remote Sensing
Wang, Q., Chen, W., Huang, Z., Tang, H., & Yang, L. (2024) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2024.3390750 · Cited by 58 · see citing works
Peng, L., Liu, X., Chen, M., Liao, W., Mao, J., & Zhou, L. (2024) · Journal of Chemical Information and Modeling · doi:10.1021/acs.jcim.4c00957 · Cited by 53 · see citing works
Novel applications of Convolutional Neural Networks in the age of Transformers
Ersavas, T., Smith, M. A., & Mattick, J. S. (2024) · Scientific Reports · doi:10.1038/s41598-024-60709-z · Cited by 50 · see citing works
Ma, S., Zhang, Y., Peng, L., Sun, C., Ding, B., & Zhu, Y. (2025) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2025.3560928 · Cited by 51 · see citing works
Self-citationWang, R., Gao, Z., Zhang, L., Yue, S., & Gao, Z. (2025) · Computer Science Review · doi:10.1016/j.cosrev.2025.100755 · Cited by 51 · see citing works
Resource-efficient Algorithms and Systems of Foundation Models: A Survey
Xu, M., Cai, D., Yin, W., Wang, S., Jin, X., & Liu, X. (2024) · ACM Computing Surveys · doi:10.1145/3706418 · Cited by 153 · see citing works
Knowledge-Empowered, Collaborative, and Co-Evolving AI Models: The Post-LLM Roadmap
Wu, F., Shen, T., Bäck, T., Chen, J., Huang, G., Jin, Y., Kuang, K., Li, M., Lu, C., Miao, J., et al. (2024) · Engineering · doi:10.1016/j.eng.2024.12.008 · Cited by 44 · see citing works
Multichannel Long-Term Streaming Neural Speech Enhancement for Static and Moving Speakers
Quan, C., & Li, X. (2024) · IEEE Signal Processing Letters · doi:10.1109/lsp.2024.3418714 · Cited by 38 · see citing works
TransRAD: Retentive Vision Transformer for Enhanced Radar Object Detection
Cheng, L., & Cao, S. (2025) · IEEE Transactions on Radar Systems · doi:10.1109/trs.2025.3537604 · Cited by 37 · see citing works
Hadamard Product in Deep Learning: Introduction, Advances and Challenges
Chrysos, G. G., Wu, Y., Pascanu, R., Torr, P. H. S., & Cevher, V. (2025) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2025.3560423 · Cited by 37 · see citing works
An outline of Prognostics and health management Large Model: Concepts, Paradigms, and challenges
Tao, L., Li, S., Liu, H., Huang, Q., Ma, L., Ning, G., Chen, Y., Wu, Y., Li, B., Zhang, W., et al. (2025) · Mechanical Systems and Signal Processing · doi:10.1016/j.ymssp.2025.112683 · Cited by 33 · see citing works
Multimodal fusion for large-scale traffic prediction with heterogeneous retentive networks
Yan, Y., Cui, S., Liu, J., Zhao, Y., Zhou, B., & Kuo, Y. (2024) · Information Fusion · doi:10.1016/j.inffus.2024.102695 · Cited by 31 · see citing works
BFA-YOLO: A balanced multiscale object detection network for building façade elements detection
Chen, Y., Wang, T., Chen, G., Zhu, K., Tan, X., Wang, J., Guo, W., Wang, Q., Luo, X., & Zhang, X. (2025) · Advanced Engineering Informatics · doi:10.1016/j.aei.2025.103289 · Cited by 31 · see citing works
Self-citationFourier-Deformable Convolution Network for Road Segmentation From Remote Sensing Images
Liu, H., Zhou, X., Wang, C., Chen, S., & Kong, H. (2024) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2024.3476087 · Cited by 28 · see citing works
Xu, Z., Chen, N., Li, G., Li, J., Zhu, H., & Zhu, Z. (2024) · Knowledge-Based Systems · doi:10.1016/j.knosys.2024.112826 · Cited by 21 · see citing works
Span-level bidirectional retention scheme for aspect sentiment triplet extraction
Yang, X., Peng, T., Bi, H., & Han, J. (2024) · Information Processing & Management · doi:10.1016/j.ipm.2024.103823 · Cited by 21 · see citing works
Luo, X., Liu, D., Kong, H., Huai, S., Chen, H., Xiong, G., & Liu, W. (2024) · ACM Transactions on Embedded Computing Systems · doi:10.1145/3701728 · Cited by 24 · see citing works
Toward a deeper understanding: RetNet viewed through Convolution
Li, C., & Zhang, C. (2024) · Pattern Recognition · doi:10.1016/j.patcog.2024.110625 · Cited by 19 · see citing works
Titans: Learning to Memorize at Test Time
Behrouz, A., Zhong, P., & Mirrokni, V. (2025) · arXiv · doi:10.48550/arxiv.2501.00663 · Cited by 24 · see citing works
“More specifically in language tasks, we compare with Transformer++ [48], RetNet [49], Gated Linear Attention (GLA) [9], Mamba [45], Mamba2 [30], DeltaNet [38], TTT [21], and Gated DeltaNet [35].”
Zhu, X., Sun, S., Lin, L., Wu, Y., & Ma, X. (2024) · Reviews in the Neurosciences · doi:10.1515/revneuro-2024-0088 · Cited by 17 · see citing works
Retentive network-based time series anomaly detection in cyber-physical systems
Min, Z., Xiao, Q., Abbas, M., & Zhang, D. (2025) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2025.110215 · Cited by 16 · see citing works
Jiang, B., Liao, M., Yun, Z., Li, G., Cheng, S., Wang, X., & Xia, Q. (2025) · Image and Vision Computing · doi:10.1016/j.imavis.2025.105463 · Cited by 20 · see citing works
Chen, G., Yang, F., Peng, W., Fan, Y., & Lyu, X. (2024) · Applied Energy · doi:10.1016/j.apenergy.2024.124266 · Cited by 15 · see citing works
Tan, M. J. T., Lichlyter, D. A., Maravilla, N. M. A. T., Schrock, W. J., Ting, F. I. L., Choa-Go, J. M., Francisco, K. K. Y., Byers, M. C., Karim, H. B. A., & AlDahoul, N. (2025) · Frontiers in Digital Health · doi:10.3389/fdgth.2025.1535018 · Cited by 18 · see citing works
The evolution of language models: From N-Grams to LLMs, and beyond
Raeini, M. (2025) · Natural Language Processing Journal · doi:10.1016/j.nlp.2025.100168 · Cited by 22 · see citing works
MV-YOLO: An Efficient Small Object Detection Framework Based on Mamba
Wu, S., Lu, X.-Y., Guo, C., & Guo, H. (2025) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2025.3584955 · Cited by 27 · see citing works
RULER: What's the Real Context Size of Your Long-Context Language Models?
Hsieh, C.-P., Sun, S., Kriman, S., Acharya, S., Rekesh, D., Jia, F., Zhang, Y., & Ginsburg, B. (2024) · arXiv · Cited by 13 · see citing works
“Finally, novel architectures (Gu et al., 2022; Fu et al., 2023a; Poli et al., 2023; Fu et al., 2023b; Sun et al., 2023a; Beck et al., 2024; Sun et al., 2024) such as Mamba (Gu & Dao, 2023) and RWKV (Peng et al., 2023) have also been proposed to efficiently handle long-context input.”
MethodFIGLUT: An Energy-Efficient Accelerator Design for FP-INT GEMM Using Look-Up Tables
Park, G., Kwon, H., Kim, J., Bae, J., Park, B., Lee, D., & Lee, Y. (2025) · IEEE International Symposium on High-Performance Computer Architecture (HPCA) · doi:10.1109/hpca61900.2025.00085 · Cited by 13 · see citing works
Graph Reconstruction Attention Fusion Network for Multimodal Sentiment Analysis
Hu, R., Yi, J., Chen, L., & Jin, Z. (2024) · IEEE Transactions on Industrial Informatics · doi:10.1109/tii.2024.3452204 · Cited by 13 · see citing works
Yuan, S., Wang, G., Mu, B., & Zhou, F. (2024) · Advances in Atmospheric Sciences · doi:10.1007/s00376-024-3313-9 · Cited by 13 · see citing works
LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compression
Jiang, H., Wu, Q., Luo, X., Li, D., Lin, C.-Y., Yang, Y., & Qiu, L. (2023) · arXiv · Cited by 12 · see citing works
“Add the subsequence x i,j = { x i , x i +1 , ..., x j } to the response y rec .”
BackgroundYou, Z., Zhang, P., Zheng, J. Y., & Cartlidge, J. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10447394 · Cited by 12 · see citing works
Beyond Attention: Breaking the Limits of Transformer Context Length with Recurrent Memory
Bulatov, A., Kuratov, Y., Kapushev, Y., & Burtsev, M. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i16.29722 · Cited by 12 · see citing works
GR-pKa: a message-passing neural network with retention mechanism for pKa prediction
Miao, R., Liu, D., Mao, L., Chen, X., Zhang, L., Yuan, Z., Shi, S., Li, H., & Li, S. (2024) · Briefings in Bioinformatics · doi:10.1093/bib/bbae408 · Cited by 12 · see citing works
Intelligent coal gangue identification: A novel amplitude frequency sensitive neural network
Zhang, Z., Zhu, Z., Meng, B., Yang, Z., Wu, M., Cheng, X., Li, B., & Liu, H. (2025) · Expert Systems with Applications · doi:10.1016/j.eswa.2025.126880 · Cited by 12 · see citing works
Sparks of Large Audio Models: A Survey and Outlook
Latif, S., Shoukat, M., Shamshad, F., Usama, M., Ren, Y., Cuayáhuitl, H., Wang, W., Zhang, X., Togneri, R., Cambria, E., et al. (2023) · arXiv · doi:10.48550/arxiv.2308.12792 · Cited by 17 · see citing works
“These techniques are efficient attention mechanisms [277– 280], length generalisation [178, 281–284], and transformer alternatives [285–288].”
BackgroundLarge language models for PHM: a review of optimization techniques and applications
Yu, T., Tang, J., Yu, Q., Li, L., Liu, Y., & Poler, R. (2025) · Autonomous Intelligent Systems · doi:10.1007/s43684-025-00100-5 · Cited by 12 · see citing works
Zhou, Z., Lu, Y., & Lv, L. (2025) · Annals of the New York Academy of Sciences · doi:10.1111/nyas.15332 · Cited by 11 · see citing works
AI for enhanced water quality data imputation: a deep learning perspective
Banjara, I. P., Poudel, S., Pariyar, K., Upreti, D., Zafeirakou, A., & Paudel, S. R. (2025) · Environmental Science Advances · doi:10.1039/d4va00367e · Cited by 10 · see citing works
Neishabouri, A., Bauer, J., Abdollahi, A., Debus, J., & Mairani, A. (2025) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2025.109777 · Cited by 10 · see citing works
HumMUSS: Human Motion Understanding Using State Space Models
Mondal, A. K., Alletto, S., & Tomè, D. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.00225 · Cited by 9 · see citing works
Transformers significantly improve splice site prediction
Jónsson, B. A., Halldorsson, G. H., Árdal, S., Rögnvaldsson, S., Einarsson, E., Sulem, P., Gudbjartsson, D. F., Melsted, P., Stefánsson, K., & Ulfarsson, M. O. (2024) · Communications Biology · doi:10.1038/s42003-024-07298-9 · Cited by 11 · see citing works
GRetNet: Gaussian Retentive Network for Hyperspectral Image Classification
Han, Z., Xu, S., Gao, L., Li, Z., & Zhang, B. (2024) · IEEE Geoscience and Remote Sensing Letters · doi:10.1109/lgrs.2024.3450536 · Cited by 8 · see citing works
Huang, Y., Zhao, W., Fu, Y., Zhu, L., & Yu, L. (2024) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2024.3475587 · Cited by 8 · see citing works
RetSeg3D: Retention-based 3D semantic segmentation for autonomous driving
Erabati, G. K., & Araújo, H. J. (2024) · Computer Vision and Image Understanding · doi:10.1016/j.cviu.2024.104231 · Cited by 8 · see citing works
Cho, S., Kim, J., & Kim, J. H. (2024) · IEEE Access · doi:10.1109/access.2024.3502219 · Cited by 8 · see citing works
HFA-Net: hierarchical feature aggregation network for micro-expression recognition
Zhang, M., Yang, W., Wang, L., Wu, Z., & Chen, D. (2025) · Complex & Intelligent Systems · doi:10.1007/s40747-025-01804-0 · Cited by 8 · see citing works
Back to recurrent processing at the crossroad of transformers and state-space models
Tiezzi, M., Casoni, M., Betti, A., Guidi, T., Gori, M., & Melacci, S. (2025) · Nature Machine Intelligence · doi:10.1038/s42256-025-01034-6 · Cited by 8 · see citing works
DouN-GNN:Double nodes graph neural network for few-shot learning
Zhang, Y., Zhou, X., Wang, N., Tang, J. H., & Tao, X. (2024) · Neurocomputing · doi:10.1016/j.neucom.2024.128970 · Cited by 7 · see citing works
Zhou, W., Kamata, S., Wang, H., Lu, P., & Zhang, M. (2024) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2024.3384403 · Cited by 7 · see citing works
RAMPGrasp: Retentive Attention-Based Multiscale Perception Grasp Detection Network
Huang, J., Liu, X., Zhu, Q. X., Wang, Y., Feng, M., Zhou, J., Zhou, Z., Chen, L., & Wang, D. (2025) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2025.3548814 · Cited by 7 · see citing works
L-Eval: Instituting Standardized Evaluation for Long Context Language Models
An, C., Gong, S., Zhong, M., Zhao, X., Li, M., Zhang, J., Kong, L., & Qiu, X. (2023) · arXiv · doi:10.48550/arxiv.2307.11088 · Cited by 6 · see citing works
LRetUNet: A U-Net-based retentive network for single-channel speech enhancement
Zhang, Y., Zhang, Z., Guo, W., Chen, W., Liu, Z., & Liu, H. (2025) · Computer Speech & Language · doi:10.1016/j.csl.2025.101798 · Cited by 6 · see citing works
Identifying different classes of geoacoustic events using machine learning
Wang, J., Zuo, Y., Dong, L., & Yan, X. (2025) · International Journal of Rock Mechanics and Mining Sciences · doi:10.1016/j.ijrmms.2025.106144 · Cited by 7 · see citing works
Self-citationA multiscale process-aware retention network for fault prediction in mixed-model production
Chen, M., Yu, R., Liang, Z., Li, K., & Qi, H. R. (2025) · Computers in Industry · doi:10.1016/j.compind.2025.104313 · Cited by 6 · see citing works
Li, Y., Liu, X., Ferreira, V. G., Balzter, H., Zhou, H., Ge, Y., Lai, M., Chu, S., Han, D., & Gu, Z. (2025) · International Journal of Applied Earth Observation and Geoinformation · doi:10.1016/j.jag.2025.104666 · Cited by 6 · see citing works
How Big Can It Get? A comparative analysis of LLMs in architecture and scaling
Yousri, R., & Safwat, S. (2023) · International Conference on Computer and Applications (ICCA) · doi:10.1109/icca59364.2023.10401818 · Cited by 6 · see citing works
Hyperspectral image classification using hybrid convolutional-based cross-patch retentive network
Arya, R. K., Peddi, R., & Srivastava, R. (2025) · Computer Vision and Image Understanding · doi:10.1016/j.cviu.2025.104382 · Cited by 6 · see citing works
Li, Y., Lai, L., Shangguan, Y., Iandola, F., Ni, Z., Chang, E., Shi, Y., & Chandra, V. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10447827 · Cited by 6 · see citing works
Lessons from the Trenches on Reproducible Evaluation of Language Models
Biderman, S., Schoelkopf, H., Sutawika, L., Gao, L., Tow, J., Abbasi, B., Aji, A. F., Ammanamanchi, P. S., Black, S., Clive, J., et al. (2024) · arXiv · doi:10.48550/arxiv.2405.14782 · Cited by 9 · see citing works
Yu, S., Yu, C., Wang, F., Zhang, B., & Tian, H. (2025) · Measurement Science and Technology · doi:10.1088/1361-6501/adf2c6 · Cited by 5 · see citing works
Huang, L., Zhang, Z., Yu, Y., & Tang, B. (2025) · IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing · doi:10.1109/jstars.2025.3586948 · Cited by 5 · see citing works
Yoon, D., Kim, T., Lee, J. W., & Rhu, M. (2024) · IEEE Computer Architecture Letters · doi:10.1109/lca.2024.3422492 · Cited by 5 · see citing works
Guo, C., Li, X., Ma, J., Li, Y., Liu, Y., Qi, H., Zhang, L., & Jin, Y. (2024) · IEEE Transactions on Fuzzy Systems · doi:10.1109/tfuzz.2024.3425664 · Cited by 5 · see citing works
PFPRNet: A Phase-Wise Feature Pyramid With Retention Network for Polyp Segmentation
Chu, J., Liu, W., Tian, Q., & Lu, W. D. (2024) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2024.3500026 · Cited by 5 · see citing works
Weird-Net: Weighted Relative Distance Attention for Efficient and Robust Sequence Processing
Hu, L., Zhan, F., Huang, W., Gan, W., Hu, H., He, H., & Han, K. (2024) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2024.3513912 · Cited by 5 · see citing works
Liquid white box model as an explainable AI for surgery
Riva-Cambrin, H. A., Singh, R., Lama, S., & Sutherland, G. R. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01769-x · Cited by 5 · see citing works
Tan, Y., Liu, Q., Liu, T., Liu, H., Wang, S.-M., & Chen, Z. (2024) · Applied Sciences · doi:10.3390/app14104259 · Cited by 5 · see citing works
Annealing Temporal–Spatial Contrastive Learning for multi-view Online Action Detection
Tan, Y., Xie, L., Jing, S., Fang, S., & Zhang, K. (2024) · Knowledge-Based Systems · doi:10.1016/j.knosys.2024.112523 · Cited by 4 · see citing works
Toward a dynamic tree-Mamba encoder for UAV tracking with vision-language
Du, G., Zhou, P., Yadikar, N., Aysa, A., & Ubul, K. (2025) · Knowledge-Based Systems · doi:10.1016/j.knosys.2025.113731 · Cited by 4 · see citing works
Spatial-Temporal Retentive Heterogeneous Graph Convolutional Network for Traffic Flow Prediction
Wang, Z., Long, B., & Xiao, J. (2024) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn60899.2024.10651032 · Cited by 4 · see citing works
AIST AIRC Submissions to the WMT23 Shared Task
Rikters, M., & Miwa, M. (2023) · Conference on Machine Translation (WMT) · doi:10.18653/v1/2023.wmt-1.13 · Cited by 4 · see citing works
A Comparative Review of Advanced Techniques for Financial Sentiment Analysis
Xiao, W. (2024) · International Conference on Modeling, Natural Language Processing and Machine Learning (CMNM) · doi:10.1145/3677779.3677791 · Cited by 4 · see citing works
ST-RetNet: A Long-Term Spatial-Temporal Traffic Flow Prediction Method
Long, B., Zhu, W., & Xiao, J. (2024) · Lecture notes in computer science · doi:10.1007/978-981-97-8502-5_1 · Cited by 4 · see citing works
Advances in edge-cloud collaboration and evolution for large-small models
Wang, Y., Shen, T., Zhang, S., Wu, F., Zhao, Z., Cai, H., Lyu, C., Ma, L., Yang, C., & Wu, F. L. (2024) · Journal of Image and Graphics · doi:10.11834/jig.240011 · Cited by 6 · see citing works
Improved YOLOv8 Method for Multi-scale Pothole Detection
Chang, J., Chen, Z., & Xia, E. (2024) · Lecture notes in computer science · doi:10.1007/978-981-97-5612-4_33 · Cited by 4 · see citing works
Toward a Deeper Understanding: Retnet Viewed Through ConvolutionPreprint
Li, C., & Zhang, C. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4637493 · Cited by 4 · see citing works
Dong, Z., Tang, T., Li, J., Zhao, W. X., & Wen, J.-R. (2023) · arXiv · doi:10.48550/arxiv.2309.13345 · Cited by 7 · see citing works
“S4 (Gu et al., 2022), DSS (Gupta et al., 2022), and GSS (Mehta et al., 2023) employed State Space Models, while RetNet (Sun et al., 2023) utilized retention mechanism, achieving parallel training and recurrent inference for long sequences.”
MethodConvolutional Retentive Network for EEG Decoding
Wang, J., Hang, W., Liang, S., Wang, Q., Chen, B., & Qin, J. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10890343 · Cited by 3 · see citing works
Self-citationXu, J., Hong, H., Zhou, Y., Hu, X., & Duan, S. (2025) · Neurocomputing · doi:10.1016/j.neucom.2025.130517 · Cited by 3 · see citing works
Khosravi, F., & Shadaram, M. (2025) · IEEE Access · doi:10.1109/access.2025.3585076 · Cited by 3 · see citing works
LS-EEND: Long-Form Streaming End-to-End Neural Diarization With Online Attractor Extraction
Liang, D., & Li, X. (2025) · IEEE Transactions on Audio Speech and Language Processing · doi:10.1109/taslpro.2025.3597446 · Cited by 3 · see citing works
Dong, Y., Wu, T., & Song, C. (2024) · Journal of Computing and Information Science in Engineering · doi:10.1115/1.4067524 · Cited by 3 · see citing works
Wei, H. L. C. L. Y. H. Y., Han, F., Liu, Q., & Han, H. (2025) · Complex System Modeling and Simulation · doi:10.23919/csms.2024.0029 · Cited by 3 · see citing works
Li, Z., Xia, T., Chang, Y., & Wu, Y. (2025) · Neurocomputing · doi:10.1016/j.neucom.2025.130711 · Cited by 3 · see citing works
Adaptive gated meta graph retention network: A model for urban traffic flow prediction
Li, X., & Bao, Y. (2025) · Expert Systems with Applications · doi:10.1016/j.eswa.2025.129703 · Cited by 3 · see citing works
Vertical slice equal sampling and transformer network for point cloud ground filtering
Wen, W., Yang, R., Tan, J., Liu, H., & Tan, J. (2024) · International Journal of Remote Sensing · doi:10.1080/01431161.2024.2368927 · Cited by 3 · see citing works
Retentive neural quantum states: efficient ansätze for ab initio quantum chemistry
Knitter, O., Zhao, D., STOKES, J. M., Ganahl, M., Leichenauer, S., & Veerapaneni, S. (2025) · Machine Learning Science and Technology · doi:10.1088/2632-2153/adcb88 · Cited by 3 · see citing works
Keep the Cost Down: A Review on Methods to Optimize LLM' s KV-Cache Consumption
Shi, L., Zhang, H., Yao, Y., Li, Z., & Zhao, H. (2024) · arXiv · doi:10.48550/arxiv.2407.18003 · Cited by 2
- Authors
- Sun, Y., Dong, L., Huang, S., Ma, S., Xia, Y., Xue, J., Wang, J., & Wei, F.
- Affiliations
- Microsoft (United States) (US)
- Tsinghua University (CN)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2023
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2307.08621
- OpenAlex
- W4384648484
- Cited by
- 142
- References
- 0
- Citations by year
- 2026: 6 · 2025: 61 · 2024: 72 · 2023: 3
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Keywords
- RetNet, large language models, recurrent neural networks, attention, retention mechanism, parallel training, long sequence modeling, linear complexity, decoding throughput, GPU memory efficiency
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
