CoChat
Sign Up
PreprintCited by 9Topic ModelingNatural Language Processing TechniquesAdvanced Neural Network Applications

Kimi Linear: An Expressive, Efficient Attention Architecture

Open access
Kimi Team, Zhang, Yu, Lin, Zongyu, Yao, Xingcheng, Hu, Jiaxi, Meng, Fanqing, Liu, Chengyin, Men, Xin +52 more·2025·arXiv
DOI 10.48550/arxiv.2510.26692arXiv 2510.26692
Open this paper in CoChat — free accountOpen PDF

Abstract

We introduce Kimi Linear, a hybrid linear attention architecture that, for the first time, outperforms full attention under fair comparisons across various scenarios -- including short-context, long-context, and reinforcement learning (RL) scaling regimes. At its core lies Kimi Delta Attention (KDA), an expressive linear attention module that extends Gated DeltaNet with a finer-grained gating mechanism, enabling more effective use of limited finite-state RNN memory. Our bespoke chunkwise algorithm achieves high hardware efficiency through a specialized variant of the Diagonal-Plus-Low-Rank (DPLR) transition matrices, which substantially reduces computation compared to the general DPLR formulation while remaining more consistent with the classical delta rule. We pretrain a Kimi Linear model with 3B activated parameters and 48B total parameters, based on a layerwise hybrid of KDA and Multi-Head Latent Attention (MLA). Our experiments show that with an identical training recipe, Kimi Linear outperforms full MLA with a sizeable margin across all evaluated tasks, while reducing KV cache usage by up to 75% and achieving up to 6 times decoding throughput for a 1M context. These results demonstrate that Kimi Linear can be a drop-in replacement for full attention architectures with superior performance and efficiency, including tasks with longer input and output lengths. To support further research, we open-source the KDA kernel and vLLM implementations, and release the pre-trained and instruction-tuned model checkpoints.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2510.26692

Cites

0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Most models mix linear attention with full attention at a fixed ratio, typically 3:1.

    “As a result, most models, such as MiniMax-M1 [94], Qwen3-Next [95], and Kimi Linear [92], mix linear attention with full attention at a fixed ratio, typically 3:1.”

    From A Survey of Large Language Models (2026) · cites this paper as [92]

    Section 7.2, “Hybrid Model,” says Kimi Linear interleaves linear and full-attention layers at a fixed 3:1 ratio and describes predefined-ratio inter-layer hybrids as widely adopted. The paper does not establish that 3:1 is typical of most models or that the other named models use that ratio. No page number is available in our copy. (LLM, conf=0.86)

    Experimental — accuracy not yet measured

  • Efficiency-focused modifications like sparse or linear attention alleviate the quadratic complexity issue but often struggle to maintain a favorable trade-off between performance and efficiency over extremely long contexts, or require substantial engineering effort to adapt for long-context deployment.

    “While efficiency-focused modifications like sparse or linear attention alleviate this issue [78,92,95], they often struggle to maintain a favorable trade-off between performance and efficiency over extremely long contexts, or require substantial engineering effort to adapt for long-context deployment.”

    From A Survey of Large Language Models (2026) · cites this paper as [92]

    Split into 2 atoms — 1 confirmed, 0 partially supported, 0 disputed, 0 misattributed, 0 unverifiable, 1 not checked. Expand the claim to see per-atom verdicts.

    Experimental — accuracy not yet measured

  • KDA refines Gated DeltaNet by making the decay channel-wise, but the scalar beta_t still carries two decisions that need not agree.

    “KDA refines Gated DeltaNet by making the decay channel-wise, but the scalar 𝛽 𝑡 in Eq. 7 still carries two decisions that need not agree.”

    From Gated DeltaNet-2: Decoupling Erase and Write in Linear Attention (2026) · cites this paper as [12]

    Section 3 states that KDA refines Gated DeltaNet’s scalar decay with a diagonal gate. The update equations in Sections 2.2 and 3 show that the scalar βₜ affects both the correction of the existing state and the addition of the new key–value association. Our copy does not state that these are two decisions that need not agree; that is the citing paper’s interpretation. (LLM, conf=0.84)

    Experimental — accuracy not yet measured

  • Gated DeltaNet-2 keeps the delta-rule fast-weight structure of GDN and KDA, but replaces the tied scalar update strength with a channel-wise erase gate b_t and a channel-wise write gate w_t.

    “Gated DeltaNet-2 keeps the delta-rule fast-weight structure of GDN and KDA, but replaces the tied scalar update strength with a channel-wise erase gate 𝑏 𝑡 and a channel-wise write gate w 𝑡 .”

    From Gated DeltaNet-2: Decoupling Erase and Write in Linear Attention (2026) · cites this paper as [12]

    Section 2.2 describes the delta rule as a fast-weight update, and Section 3 shows that KDA retains a delta-rule update while adding fine-grained decay gating. The copy does not describe Gated DeltaNet-2 or say that it replaces a tied scalar update strength with separate channel-wise erase and write gates, b_t and w_t (pages unavailable). (LLM, conf=0.78)

    Experimental — accuracy not yet measured

  • Gated DeltaNet-2 removes the scalar beta_t tie in Gated DeltaNet and KDA.

    “This removes the scalar 𝛽 𝑡 tie in Gated DeltaNet and KDA, recovers both as special cases, and preserves efficient chunkwise training through a WYform with gate-aware kernels.”

    From Gated DeltaNet-2: Decoupling Erase and Write in Linear Attention (2026) · cites this paper as [12]

    The Gated DeltaNet as Weight Decay and §3 Kimi Delta Attention passages show β_t multiplying both the corrective term and the new key–value term in GDN and KDA. Our copy does not describe Gated DeltaNet-2 or say that it removes this tie, recovers both models as special cases, or uses gate-aware WY-form kernels. No page numbers are available in the indexed passages. (LLM, conf=0.82)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

The publisher hasn’t deposited this paper’s reference list.

Research works printed but not linked: 109

  1. Sandhini Agarwal et al. "gpt-oss-120b & gpt-oss-20b model card". In: arXiv preprint arXiv:2508.10925 (2025). · arXiv
  2. Joshua Ainslie et al. "Colt5: Faster long-range transformers with conditional computation". In: arXiv preprint arXiv:2303.09752 (2023). · arXiv
  3. Simran Arora et al. "Simple linear attention language models balance the recall-throughput tradeoff". In: Forty-first International Conference on Machine Learning. 2024. URL: https://openreview.net/forum? id=e93ffDcpH3.
  4. Simran Arora et al. Zoology: Measuring and Improving Recall in Efficient Language Models. 2023. arXiv: 2312.04927 [cs.CL]. · arXiv
  5. Yushi Bai et al. "Longbench v2: Towards deeper understanding and reasoning on realistic long-context multi- tasks". In: arXiv preprint arXiv:2412.15204 (2024). · arXiv
  6. Federico Barbero et al. "Round and Round We Go! What makes Rotary Positional Encodings useful?" In: Proceedings of ICLR. 2025. URL: https://openreview.net/forum?id=GtvuNrk58a.
  7. Amanda Bertsch et al. "Unlimiformer: Long-range transformers with unlimited length input". In: Advances in NeurIPS 36 (2023), pp. 35522-35543.
  8. Stella Biderman et al. "Lessons from the trenches on reproducible evaluation of language models". In: arXiv preprint arXiv:2405.14782 (2024). · arXiv
  9. Aaron Blakeman et al. "Nemotron-h: A family of accurate and efficient hybrid mamba-transformer models". In: arXiv preprint arXiv:2504.03624 (2025). · arXiv
  10. Egor Bogomolov et al. "Long code arena: a set of benchmarks for long-context code models". In: arXiv preprint arXiv:2406.11612 (2024). · arXiv
  11. Peter Clark et al. "Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge". In: arXiv:1803.05457v1 (2018). · arXiv
  12. Ganqu Cui et al. "The entropy mechanism of reinforcement learning for reasoning language models". In: arXiv preprint arXiv:2505.22617 (2025). · arXiv
  13. Tri Dao et al. "FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness". In: Advances in NeurIPS. 2022, pp. 16344-16359. URL: https://proceedings.neurips.cc/paper_files/paper/ 2022/file/67d57c32e20fd0a7a302cb81d36e40d5-Paper-Conference.pdf.
  14. DeepSeek-AI et al. DeepSeek-V3 Technical Report. 2025. arXiv: 2412.19437 [cs.CL]. URL: https:// arxiv.org/abs/2412.19437. · arXiv
  15. Jiayu Ding et al. LongNet: Scaling Transformers to 1,000,000,000 Tokens. 2023. arXiv: 2307.02486 [cs.CL]. URL: https://arxiv.org/abs/2307.02486. · arXiv
  16. Juechu Dong et al. Flex Attention: A Programming Model for Generating Optimized Attention Kernels. 2024. arXiv: 2412.05496 [cs.LG]. URL: https://arxiv.org/abs/2412.05496. · arXiv
  17. Xin Dong et al. Hymba: A Hybrid-head Architecture for Small Language Models. 2024. arXiv: 2411.13676 [cs.CL]. URL: https://arxiv.org/abs/2411.13676. · arXiv
  18. Jusen Du et al. "Mom: Linear sequence modeling with mixture-of-memories". In: arXiv preprint arXiv:2502.13685 (2025). · arXiv
  19. Jusen Du et al. "Native Hybrid Attention for Efficient Sequence Modeling". In: arXiv preprint arXiv:2510.07019 (2025). · arXiv
  20. Tianyu Fu et al. "Moa: Mixture of sparse attention for automatic large language model compression". In: arXiv preprint arXiv:2406.14909 (2024). · arXiv
  21. Aryo Pradipta Gema et al. "Are we done with mmlu?" In: arXiv preprint arXiv:2406.04127 (2024). · arXiv
  22. Riccardo Grazzi et al. "Unlocking State-Tracking in Linear RNNs Through Negative Eigenvalues". In: Proceed- ings of ICLR. 2025. URL: https://openreview.net/forum?id=UvTo3tVBk2.
  23. Albert Gu and Tri Dao. Mamba: Linear-Time Sequence Modeling with Selective State Spaces. 2023. arXiv: 2312.00752 [cs.LG]. · arXiv
  24. Albert Gu, Karan Goel, and Christopher Ré. Efficiently Modeling Long Sequences with Structured State Spaces. 2022. arXiv: 2111.00396 [cs.LG]. · arXiv
  25. Xiangming Gu et al. When Attention Sink Emerges in Language Models: An Empirical View. 2025. arXiv: 2410.10781 [cs.CL]. URL: https://arxiv.org/abs/2410.10781. · arXiv
  26. Yuxian Gu et al. Jet-Nemotron: Efficient Language Model with Post Neural Architecture Search. 2025. arXiv: 2508.15884 [cs.CL]. URL: https://arxiv.org/abs/2508.15884. · arXiv
  27. Daya Guo et al. "DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning". In: Nature 645.8081 (2025), pp. 633-638.
  28. Han Guo et al. "Log-linear attention". In: arXiv preprint arXiv:2506.04761 (2025). · arXiv
  29. Qipeng Guo et al. "Star-transformer". In: arXiv preprint arXiv:1902.09113 (2019). · arXiv
  30. Dan Hendrycks et al. Measuring Massive Multitask Language Understanding. 2021. arXiv: 2009.03300 [cs.CY]. URL: https://arxiv.org/abs/2009.03300. · arXiv
  31. Jordan Hoffmann et al. Training Compute-Optimal Large Language Models. 2022. arXiv: 2203 . 15556 [cs.CL]. URL: https://arxiv.org/abs/2203.15556. · arXiv
  32. Jiaxi Hu et al. "Attractor memory for long-term time series forecasting: A chaos perspective". In: Advances in NeurIPS 37 (2024), pp. 20786-20818.
  33. Jiaxi Hu et al. "Comba: Improving Nonlinear RNNs with Closed-loop Control". In: arXiv preprint arXiv:2506.02475 (2025). · arXiv
  34. Ermo Hua et al. "Fourier Position Embedding: Enhancing Attention's Periodic Extension for Length General- ization". In: arXiv preprint arXiv:2412.17739 (2024). · arXiv
  35. Weizhe Hua et al. "Transformer Quality in Linear Time". In: Proceedings of ICML. Ed. by Kamalika Chaudhuri et al. PMLR, 2022, pp. 9099-9117. URL: https://proceedings.mlr.press/v162/hua22a.html.
  36. Yuzhen Huang et al. "C-eval: A multi-level multi-discipline chinese evaluation suite for foundation models". In: Advances in NeurIPS 36 (2023), pp. 62991-63010.
  37. Naman Jain et al. "Livecodebench: Holistic and contamination free evaluation of large language models for code". In: arXiv preprint arXiv:2403.07974 (2024). · arXiv
  38. Samy Jelassi et al. Repeat After Me: Transformers are Better than State Space Models at Copying. 2024. arXiv: 2402.01032 [cs.LG]. · arXiv
  39. Mandar Joshi et al. "Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension". In: arXiv preprint arXiv:1705.03551 (2017). · arXiv
  40. Angelos Katharopoulos et al. "Transformers are RNNs: Fast Autoregressive Transformers with Linear Atten- tion". In: Proceedings of ICML. Ed. by Hal Daumé III and Aarti Singh. PMLR, 2020, pp. 5156-5165. URL: https://proceedings.mlr.press/v119/katharopoulos20a.html.
  41. Amirhossein Kazemnejad et al. "The impact of positional encoding on length generalization in transformers". In: Advances in NeurIPS 36 (2023), pp. 24892-24928.
  42. Team Kimi et al. "Kimi k2: Open agentic intelligence". In: arXiv preprint arXiv:2507.20534 (2025). · arXiv
  43. Nikita Kitaev, Łukasz Kaiser, and Anselm Levskaya. "Reformer: The efficient transformer". In: arXiv preprint arXiv:2001.04451 (2020). · arXiv
  44. Satyapriya Krishna et al. "Fact, fetch, and reason: A unified evaluation of retrieval-augmented generation". In: arXiv preprint arXiv:2409.12941 (2024). · arXiv
  45. Hanyu Lai et al. "A Survey of Post-Training Scaling in Large Language Models". In: Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2025, pp. 2771- 2791.
  46. Disen Lan et al. "Liger: Linearizing Large Language Models to Gated Recurrent Structures". In: arXiv preprint arXiv:2503.01496 (2025). · arXiv
  47. Yixing Li et al. "Transmamba: Flexibly switching between transformer and mamba". In: arXiv preprint arXiv:2503.24067 (2025). · arXiv
  48. Opher Lieber et al. Jamba: A Hybrid Transformer-Mamba Language Model. 2024. arXiv: 2403 . 19887 [cs.CL].
  49. Zhixuan Lin et al. "Forgetting transformer: Softmax attention with a forget gate". In: arXiv preprint arXiv:2503.02130 (2025). · arXiv
  50. Bo Liu et al. "Longhorn: State Space Models are Amortized Online Learners". In: ArXiv abs/2407.14207 (2024). URL: https://api.semanticscholar.org/CorpusID:271310065. · arXiv
  51. Jiawei Liu et al. "Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation". In: Thirty-seventh Conference on NeurIPS. 2023. URL: https://openreview. net/forum?id=1qvx610Cu7.
  52. Jiawei Liu et al. "Repoqa: Evaluating long context code understanding". In: arXiv preprint arXiv:2406.06025 (2024). · arXiv
  53. Jingyuan Liu et al. Muon is Scalable for LLM Training. 2025. arXiv: 2502.16982 [cs.LG]. URL: https: //arxiv.org/abs/2502.16982. · arXiv
  54. Enzhe Lu et al. MoBA: Mixture of Block Attention for Long-Context LLMs. 2025. arXiv: 2502.13189 [cs.LG]. URL: https://arxiv.org/abs/2502.13189. · arXiv
  55. William Merrill, Jackson Petty, and Ashish Sabharwal. "The illusion of state in state-space models". In: arXiv preprint arXiv:2404.08819 (2024). · arXiv
  56. MiniMax et al. MiniMax-01: Scaling Foundation Models with Lightning Attention. 2025. arXiv: 2501.08313 [cs.CL]. · arXiv
  57. Tsendsuren Munkhdalai, Manaal Faruqui, and Siddharth Gopal. Leave No Context Behind: Efficient Infinite Context Transformers with Infini-attention. 2024. arXiv: 2404.07143 [cs.CL]. · arXiv
  58. Tsendsuren Munkhdalai and Adam Trischler. Metalearning with Hebbian Fast Weights. 2018. arXiv: 1807. 05076 [cs.NE]. URL: https://arxiv.org/abs/1807.05076. · arXiv
  59. Tsendsuren Munkhdalai et al. "Metalearned Neural Memory". In: ArXiv abs/1907.09720 (2019). URL: https: //api.semanticscholar.org/CorpusID:198179407. · arXiv
  60. Long Ouyang et al. "Training language models to follow instructions with human feedback". In: Advances in NeurIPS 35 (2022), pp. 27730-27744.
  61. Bo Peng et al. RWKV-7 "Goose" with Expressive Dynamic State Evolution. 2025. arXiv: 2503.14456 [cs.CL]. · arXiv
  62. Bowen Peng et al. "Yarn: Efficient context window extension of large language models". In: arXiv preprint arXiv:2309.00071 (2023). · arXiv
  63. Piotr Piękos, Róbert Csordás, and Jürgen Schmidhuber. "Mixture of Sparse Attention: Content-Based Learnable Sparse Attention via Expert-Choice Routing". In: arXiv preprint arXiv:2505.00315 (2025). · arXiv
  64. Aske Plaat et al. "Reasoning with large language models, a survey". In: CoRR (2024).
  65. Ofir Press, Noah Smith, and Mike Lewis. "Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation". In: Proceedings of ICLR. 2022. URL: https://openreview.net/forum?id= R8sQPpGCv0.
  66. Krishna C. Puvvada et al. SWAN-GPT: An Efficient and Scalable Approach for Long-Context Language Modeling. 2025. arXiv: 2504.08719 [cs.CL]. · arXiv
  67. Zhen Qin et al. HGRN2: Gated Linear RNNs with State Expansion. 2024. arXiv: 2404.07904 [cs.CL]. · arXiv
  68. Zhen Qin et al. TransNormerLLM: A Faster and Better Large Language Model with Improved TransNormer. 2024. arXiv: 2307.14995 [cs.CL]. · arXiv
  69. Zihan Qiu et al. Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Free. 2025. arXiv: 2505.06708 [cs.CL]. · arXiv
  70. Xiaoye Qu et al. "A survey of efficient reasoning for large reasoning models: Language, multimodality, and beyond". In: arXiv preprint arXiv:2503.21614 (2025). · arXiv
  71. David Rein et al. "Gpqa: A graduate-level google-proof q&a benchmark". In: First Conference on Language Modeling. 2024.
  72. Keisuke Sakaguchi et al. WinoGrande: An Adversarial Winograd Schema Challenge at Scale. 2019. arXiv: 1907.10641 [cs.CL]. URL: https://arxiv.org/abs/1907.10641. · arXiv
  73. Imanol Schlag, Kazuki Irie, and Jürgen Schmidhuber. "Linear Transformers Are Secretly Fast Weight Program- mers". In: Proceedings of ICML. Ed. by Marina Meila and Tong Zhang. PMLR, 2021, pp. 9355-9366. URL: https://proceedings.mlr.press/v139/schlag21a.html.
  74. Imanol Schlag, Tsendsuren Munkhdalai, and Jürgen Schmidhuber. Learning Associative Inference Using Fast Weight Memory. 2021. arXiv: 2011.07831 [cs.LG]. URL: https://arxiv.org/abs/2011.07831. · arXiv
  75. Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. Self-Attention with Relative Position Representations. 2018. arXiv: 1803.02155 [cs.CL]. · arXiv
  76. Weigao Sun et al. Speed Always Wins: A Survey on Efficient Architectures for Large Language Models. 2025. arXiv: 2508.09834 [cs.CL]. URL: https://arxiv.org/abs/2508.09834. · arXiv
  77. Yu Sun et al. "Learning to (Learn at Test Time): RNNs with Expressive Hidden States". In: ArXiv abs/2407.04620 (2024). URL: https://api.semanticscholar.org/CorpusID:271039606. · arXiv
  78. Yutao Sun et al. "Efficient attention mechanisms for large language models: A survey". In: arXiv preprint arXiv:2507.19595 (2025). · arXiv
  79. Yutao Sun et al. Retentive Network: A Successor to Transformer for Large Language Models. 2023. arXiv: 2307.08621 [cs.CL]. · arXiv
  80. Yutao Sun et al. You Only Cache Once: Decoder-Decoder Architectures for Language Models. 2024. arXiv: 2405.05254 [cs.CL]. URL: https://arxiv.org/abs/2405.05254. · arXiv
  81. Mirac Suzgun et al. "Challenging big-bench tasks and whether chain-of-thought can solve them". In: arXiv preprint arXiv:2210.09261 (2022). · arXiv
  82. Kimi Team et al. Kimi k1.5: Scaling Reinforcement Learning with LLMs. 2025. arXiv: 2501.12599 [cs.AI]. URL: https://arxiv.org/abs/2501.12599. · arXiv
  83. MiniCPM Team et al. MiniCPM4: Ultra-Efficient LLMs on End Devices. 2025. arXiv: 2506.07900 [cs.CL]. URL: https://arxiv.org/abs/2506.07900. · arXiv
  84. Tencent Hunyuan Team et al. "Hunyuan-turbos: Advancing large language models through mamba-transformer synergy and adaptive chain-of-thought". In: arXiv preprint arXiv:2505.15431 (2025). · arXiv
  85. Hugo Touvron et al. LLaMA: Open and Efficient Foundation Language Models. 2023. arXiv: 2302.13971 [cs.CL]. · arXiv
  86. Ashish Vaswani et al. "Attention is All you Need". In: Advances in NeurIPS. Ed. by I. Guyon et al. Curran Associates, Inc., 2017. URL: https://proceedings.neurips.cc/paper_files/paper/2017/file/ 3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf.
  87. Roger Waleffe et al. An Empirical Study of Mamba-based Language Models. 2024. arXiv: 2406 . 07887 [cs.LG]. URL: https://arxiv.org/abs/2406.07887. · arXiv
  88. Sinong Wang et al. Linformer: Self-Attention with Linear Complexity. 2020. arXiv: 2006.04768 [cs.LG]. · arXiv
  89. Yubo Wang et al. "Mmlu-pro: A more robust and challenging multi-task language understanding benchmark". In: Advances in NeurIPS 37 (2024), pp. 95266-95290.
  90. Kaiyue Wen, Xingyu Dang, and Kaifeng Lyu. "Rnns are not transformers (yet): The key bottleneck on in-context retrieval". In: arXiv preprint arXiv:2402.18510 (2024). · arXiv
  91. Colin White et al. "Livebench: A challenging, contamination-free llm benchmark". In: arXiv preprint arXiv:2406.19314 4 (2024). · arXiv
  92. Yuhuai Wu et al. "Memorizing transformers". In: arXiv preprint arXiv:2203.08913 (2022). · arXiv
  93. Guangxuan Xiao et al. "Efficient streaming language models with attention sinks". In: arXiv preprint arXiv:2309.17453 (2023). · arXiv
  94. Wenhan Xiong et al. Effective Long-Context Scaling of Foundation Models. 2023. arXiv: 2309.16039 [cs.CL]. URL: https://arxiv.org/abs/2309.16039. · arXiv
  95. Ruyi Xu et al. "Xattention: Block sparse attention with antidiagonal scoring". In: arXiv preprint arXiv:2503.16428 (2025). · arXiv
  96. Bowen Yang et al. Rope to Nope and Back Again: A New Hybrid Attention Strategy. 2025. arXiv: 2501.18795 [cs.CL]. URL: https://arxiv.org/abs/2501.18795. · arXiv
  97. Songlin Yang, Jan Kautz, and Ali Hatamizadeh. "Gated Delta Networks: Improving Mamba2 with Delta Rule". In: Proceedings of ICLR. 2025. URL: https://openreview.net/forum?id=r8H7xhYPwz.
  98. Songlin Yang and Bailin Wang. "Parallelizing Linear Transformers with the Delta Rule over Sequence Length". In: ArXiv abs/2406.06484 (2024). URL: https://api.semanticscholar.org/CorpusID:270371554. · arXiv
  99. Songlin Yang et al. "Gated Linear Attention Transformers with Hardware-Efficient Training". In: Proceedings of ICML. PMLR, 2024.
  100. Songlin Yang et al. "PaTH Attention: Position Encoding via Accumulating Householder Transformations". In: arXiv preprint arXiv:2505.16381 (2025). · arXiv
  101. Morris Yau et al. "Sequential-Parallel Duality in Prefix Scannable Models". In: arXiv preprint arXiv:2506.10918 (2025). · arXiv
  102. Jingyang Yuan et al. Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention. 2025. arXiv: 2502.11089 [cs.CL]. URL: https://arxiv.org/abs/2502.11089. · arXiv
  103. Manzil Zaheer et al. "Big bird: Transformers for longer sequences". In: Advances in NeurIPS 33 (2020), pp. 17283-17297.
  104. Rowan Zellers et al. "HellaSwag: Can a Machine Really Finish Your Sentence?" In: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. 2019.
  105. Biao Zhang and Rico Sennrich. "Root mean square layer normalization". In: Advances in NeurIPS 32 (2019).
  106. Michael Zhang et al. "Lolcats: On low-rank linearizing of large language models". In: arXiv preprint arXiv:2410.10254 (2024). · arXiv
  107. Yu Zhang et al. Gated Slot Attention for Efficient Linear-Time Sequence Modeling. 2024. arXiv: 2409.07146 [cs.CL]. · arXiv
  108. Shu Zhong et al. "Understanding Transformer from the Perspective of Associative Memory". In: arXiv preprint arXiv:2505.19488 (2025). · arXiv
  109. Jingwei Zuo et al. Falcon-H1: A Family of Hybrid-Head Language Models Redefining Efficiency and Perfor- mance. 2025. arXiv: 2507.22448 [cs.CL]. URL: https://arxiv.org/abs/2507.22448. · arXiv

Cited by

Sorted by citations

Related

Authors
Kimi Team, Zhang, Y., Lin, Z., Yao, X., Hu, J., Meng, F., Liu, C., Men, X., Yang, S., Li, Z., Li, W., Lu, E., Liu, W., Chen, Y., Xu, W., Yu, L., Wang, Y., Fan, Y., Zhong, L., Yuan, E., Zhang, D., Zhang, Y., Liu, T. Y., Wang, H., Fang, S., He, W., Liu, S., Li, Y., Su, J., Qiu, J., Pang, B., Yan, J., Jiang, Z., Huang, W., Yin, B., You, J., Wei, C., Wang, Z., Hong, C., Chen, Y., Chen, G., Wang, Y., Zheng, H., Wang, F., Liu, Y., Dong, M., Zhang, Z., Pan, S., Wu, W., Wu, Y., Guan, L., Tao, J., Fu, G., Xu, X., Wang, Y., Lai, G., Wu, Y., Zhou, X., Yang, Z., & Du, Y.
ORCID
Venue
arXiv
Publisher
arXiv
Published
2025
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2510.26692
arXiv
2510.26692
OpenAlex
W7103188877
Cited by
1
References
0
Citations by year
2026: 1
Type
posted content
Language
en
Open access
Licence
Locations
  • arXiv (Cornell University) (submittedVersion, cc-by-nc-nd)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
Keywords
linear attention, long-context decoding, multi-head latent attention, decoding throughput

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review