OjaKV: Context-Aware Online Low-Rank KV Cache Compression
Open accessAbstract
The expanding long-context capabilities of large language models are constrained by a significant memory bottleneck: the key-value (KV) cache required for autoregressive generation. This bottleneck is substantial; for instance, a Llama-3.1-8B model processing a 32K-token prompt at a batch size of 4 requires approximately 16GB for its KV cache, a size exceeding the model's weights. While KV-cache compression via low-rank projection is a promising direction, existing methods rely on a static, offline-learned subspace that performs poorly under data distribution shifts. To overcome these limitations, we introduce OjaKV, a novel framework that integrates a strategic hybrid storage policy with online subspace adaptation. First, OjaKV recognizes that not all tokens are equally important for compression; it preserves the crucial first and most recent tokens in full-rank, maintaining high-fidelity anchors for attention. Second, for the vast majority of intermediate tokens, it applies low-rank compression by incrementally adapting the projection basis using Oja's algorithm for online principal component analysis. This adaptation involves a comprehensive update during prompt prefilling and lightweight periodic updates during decoding, ensuring the subspace remains aligned with the evolving context. Crucially, our framework is fully compatible with modern attention modules like FlashAttention. Experiments demonstrate that OjaKV maintains or even improves zero-shot accuracy at high compr…
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
DynaKV learns a gating network by fine-tuning, while OjaKV is training-free but allocates rank by position.
“Two methods vary rank per token : DynaKV (Lu et al. 2026) learns a gating network by fine-tuning, while OjaKV (Zhu et al. 2025) is training-free but allocates by position-full-rank anchors and one low rank for the rest, over an online Oja-adapted basis.”
From VarRate: Training-Free Variable-Rate KV Cache Compression for Long-Context LLMs (2026) · cites this paper as Zhu et al. 2025
The Abstract (p. 1) says OjaKV requires no model fine-tuning, and the Methodology (p. 4) describes full-rank retention for selected tokens and online-adapted low-rank compression for the rest. But §5.1 (p. 5) says tokens are selected by reconstruction error, explicitly contrasting this with positional heuristics. The position-based allocation attributed to OjaKV is therefore incorrect. (LLM, conf=0.90)
Experimental — accuracy not yet measuredVarRate is the first training-free method to set the per-token rank from a query-salience signal-a graded budget, not a fixed positional split.
“To our knowledge, VarRate is the first training-free method to set the per-token rank from a query-salience signal-a graded budget, not a fixed positional split.”
From VarRate: Training-Free Variable-Rate KV Cache Compression for Long-Context LLMs (2026) · cites this paper as Zhu et al. 2025
Section 5.1 (p. 5) describes query-weighted error scores and a choice between retaining selected tokens at full rank and compressing the rest, rather than using a fixed positional rule. It does not discuss VarRate, establish that VarRate is first, or describe a graded per-token rank budget. (LLM, conf=0.82)
Experimental — accuracy not yet measuredComparable adaptive-rank codecs reach this allocation only through training; VarRate requires none.
“Comparable adaptive-rank codecs reach this allocation only through training; VarRate requires none.”
From VarRate: Training-Free Variable-Rate KV Cache Compression for Long-Context LLMs (2026) · cites this paper as Zhu et al. 2025
Section 2.2 (p. 3) says Multi-Head Latent Attention requires training new models, while OjaKV adapts its projection matrices during inference. Appendix A.3 (p. 13) describes rank selection from calibration data, but neither passage addresses comparable adaptive-rank codecs achieving the claimed allocation only through training or whether VarRate requires training. Searches for adaptive-rank allocation and VarRate found no passage that settles the comparison. (LLM, conf=0.74)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
The publisher hasn’t deposited this paper’s reference list.
Research works printed but not linked: 3
- Ming Gu and Stanley C Eisenstat. 1994. A stable and efficient algorithm for the rank-one modification of the symmetric eigenproblem. SIAM journal on Matrix Analysis and Applications , 15(4):1266-1276.
- Mandy Guo, Zihang Dai, Denny Vrandeˇ ci´ c, and Rami Al-Rfou. 2020. Wiki-40b: Multilingual language model dataset. In Proceedings of the Twelfth Language Resources and Evaluation Conference , pages 2440-2452.
- Yuhong Li, Yingbing Huang, Bowen Yang, Bharat Venkitesh, Acyr Locatelli, Hanchen Ye, Tianle Cai, Patrick Lewis, and Deming Chen. 2024. Snapkv: Llm knows what you are looking for before generation. Advances in Neural Information Processing Systems , 37:22947-22970.
Cited by
Sorted by citations
VarRate: Training-Free Variable-Rate KV Cache Compression for Long-Context LLMs
Esmat, S., Shah, D., & Jannesari, A. (2026) · arXiv · doi:10.48550/arxiv.2607.15498
“Methods that do vary rank per token either train a gate for it (Lu et al. 2026) or split it by fixed position (Zhu et al. 2025)—never from the model’s own attention.”
- Authors
- Zhu, Y., Yang, D. H., Amiri, M. M., Murugesan, K., Pedapati, T., & Chen, P.-Y.
- Affiliations
- Rensselaer Polytechnic Institute (US)
- IBM (United States) (US)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2025
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2509.21623
- OpenAlex
- W7083684258
- Cited by
- 0
- References
- 0
- Type
- posted content
- Language
- en
- Open access
- green
- doi.org
- Locations
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Keywords
- key-value cache compression, low-rank projection, online principal component analysis, long-context inference, memory efficiency, large language models, FlashAttention
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
