Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention
Open accessAbstract
Long-context modeling is crucial for next-generation language models, yet the high computational cost of standard attention mechanisms poses significant computational challenges. Sparse attention offers a promising direction for improving efficiency while maintaining model capabilities. We present NSA, a Natively trainable Sparse Attention mechanism that integrates algorithmic innovations with hardware-aligned optimizations to achieve efficient long-context modeling. NSA employs a dynamic hierarchical sparse strategy, combining coarse-grained token compression with fine-grained token selection to preserve both global context awareness and local precision. Our approach advances sparse attention design with two key innovations: (1) We achieve substantial speedups through arithmetic intensity-balanced algorithm design, with implementation optimizations for modern hardware. (2) We enable end-to-end training, reducing pretraining computation without sacrificing model performance. As shown in Figure 1, experiments show the model pretrained with NSA maintains or exceeds Full Attention models across general benchmarks, long-context tasks, and instruction-based reasoning. Meanwhile, NSA achieves substantial speedups over Full Attention on 64k-length sequences across decoding, forward propagation, and backward propagation, validating its efficiency throughout the model lifecycle.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Existing context-compression methods often use distillation to produce an offline prompt, memory, or compressed KV representation before generation.
“Existing context-compression methods often use distillation to produce an offline prompt, memory, or compressed KV representation before generation [7, 28, 53, 32, 19, 34, 56, 30, 31].”
From Learning What to Remember: Test-Time Training via Context Distillation (2026) · cites this paper as [53]
Section 3.3.1 (p. 6) describes producing compressed key and value representations, supporting the compressed-KV part of the claim. It does not describe distillation or an offline prompt or memory produced before generation; Section 7.2 (p. 16) instead discusses dynamic KV-cache pruning during inference. (LLM, conf=0.83)
Experimental — accuracy not yet measuredModern architectures that approximate full attention with a sliding window or stack attention and RNN layers together are still less effective than full attention in using longer context to achieve better performance in language modeling.
“Some modern architectures approximate full attention with a sliding window [1, 107], or stack attention and RNN layers together [91, 11]. However, these techniques are still less effective than full attention in using longer context to achieve better performance in language modeling.”
From End-to-End Test-Time Training for Long Context (2025) · cites this paper as [107]
Section 7.1 (p. 16) identifies sliding-window attention as a fixed-window approach and contrasts it with NSA’s ability to use the full context. The paper does not establish that sliding-window or combined attention–RNN architectures perform worse than full attention in language modeling; its Conclusion (p. 17) reports that NSA matches or exceeds full-attention baselines. (LLM, conf=0.80)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
This paper’s reference list could not be split into single entries.
LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Models
Jiang, H., Wu, Q., Lin, C.-Y., Yang, Y., & Qiu, L. (2023) · arXiv · Cited by 168 · cited 1× · see citing works
“…demonstrate this potential through diverse strategies: KV-cache eviction methods (Zhang et al., 2023b; Li et al., 2024; Zhou et al., 2024), blockwise KV-cache selection methods (Tang et al., 2024; Xiao et al., 2024a; Gao et al., 2024), and sampling, clustering or hashing-based selection…”
Flagged first, then by citations
Cited by
Sorted by citations
DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models
DeepSeek-AI, Liu, A., Mei, A., Lin, B., Xue, B., Wang, B., Xu, B., Wu, B., Zhang, B., Lin, C., et al. (2025) · arXiv · doi:10.48550/arxiv.2512.02556 · Cited by 19 · see citing works
Self-citationEnd-to-End Test-Time Training for Long Context
Tandon, A., Dalal, K., Li, X., Koceja, D., Rød, M., Buchanan, S., Wang, X., Leskovec, J., Koyejo, S., Hashimoto, T., et al. (2025) · arXiv
“Some modern architectures approximate full attention with a sliding window [1, 107], or stack attention and RNN layers together [91, 11].”
IndexMem: Learned KV-Cache Eviction with Latent Memory for Long-Context LLM Inference
Yang, X., Gu, H., Xu, B., Li, L., Liu, B., Liu, J., Zhu, Q., Guo, Y., & Han, S. (2026) · arXiv · doi:10.48550/arxiv.2605.25475
“NSA designs natively trainable sparse attention mechanisms aligned with modern hardware, emphasizing both training viability and inference efficiency (Yuan et al., 2025b).”
Learning What to Remember: Test-Time Training via Context Distillation
Wang, Z., Dang, X., Zhu, R.-J., Wen, Z., Fu, H., Chai, W., & Lee, J. D. (2026) · arXiv · doi:10.48550/arxiv.2608.01672
“Existing context-compression methods often use distillation to produce an offline prompt, memory, or compressed KV representation before generation [7, 28, 53, 32, 19, 34, 56, 30, 31].”
Zhang, J., & Bottou, L. (2025) · arXiv · doi:10.48550/arxiv.2507.03285
- Authors
- Yuan, J., Gao, H., Dai, D., Luo, J., Zhao, L., Zhang, Z., Xie, Z., Wei, Y. X., Wang, L., Xiao, Z., Wang, Y., Ruan, C., Zhang, M., Liang, W., & Zeng, W.
- ORCID
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2025
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2502.11089
- OpenAlex
- W4407690159
- Cited by
- 4
- References
- 1
- Citations by year
- 2026: 2 · 2025: 2
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- sparse attention, long-context modeling, token compression, token selection, hierarchical sparsity, arithmetic intensity, end-to-end learning, efficient language models
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
