KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache
Open accessAbstract
Efficiently serving large language models (LLMs) requires batching of many requests to reduce the cost per request. Yet, with larger batch sizes and longer context lengths, the key-value (KV) cache, which stores attention keys and values to avoid re-computations, significantly increases memory demands and becomes the new bottleneck in speed and memory usage. Additionally, the loading of the KV cache causes the computational core to be idle, which limits the inference speed. A straightforward and effective solution to reduce KV cache size is quantization, which decreases the total bytes taken by KV cache. However, there is a lack of in-depth studies that explore the element distribution of KV cache to understand the hardness and limitation of KV cache quantization. To fill the gap, we conducted a comprehensive study on the element distribution in KV cache of popular LLMs. Our findings indicate that the key cache should be quantized per-channel, i.e., group elements along the channel dimension and quantize them together. In contrast, the value cache should be quantized per-token. From this analysis, we developed a tuning-free 2bit KV cache quantization algorithm named KIVI. With hardware-friendly implementation, KIVI can enable Llama, Falcon, and Mistral models to maintain almost the same quality while using $\mathbf{2.6\times}$ less peak memory (including model weight). This reduction in memory usage enables up to $\mathbf{4\times}$ larger batch size, bringing $\mathbf{2.35\ti…
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Cache quantization compresses all entries to 2-4 bits.
“Cache quantization compresses all entries to 2-4 bits (Liu et al., 2024; Hooper et al., 2024).”
From Tamarin (HSQ): Reversible Hierarchical KV-Cache Compression for LLM Inference (2026) · cites this paper as Liu et al., 2024
The Introduction (p. 2) describes 2-bit KIVI quantization, and the preliminary study (p. 3) evaluates both 2-bit and 4-bit KV-cache quantization. But the Introduction (p. 2) says KIVI quantizes only the grouped cache and keeps residual key and value entries in full precision, so the source does not support “all entries.” (LLM, conf=0.90)
Experimental — accuracy not yet measuredKeys participate in the softmax attention score computation and degrade rapidly under low-bit-width quantization, while Values are more robust to lossy compression.
“Keys participate in the softmax attention score computation and degrade rapidly under low-bit-width quantization Liu et al. [2024], Hooper et al. [2024]. Values are more robust to lossy compression.”
From PolyKV: Shared Asymmetric TurboQuant-Compressed KV Memory for Multi-Agent LLMs (2026) · cites this paper as Liu et al. [2024]
Section 3.2 (p. 4) shows that keys enter the softmax attention-score calculation and that per-token key quantization produces much larger attention-score error than per-channel quantization. But the paper does not say keys generally degrade rapidly at low bit widths: it reports a small accuracy drop with 2-bit per-channel keys. Its finding that values tolerate 2-bit per-token quantization also does not extend to per-channel quantization, which significantly worsens accuracy (Section 3.1, p. 3). (LLM, conf=0.86)
Experimental — accuracy not yet measuredQuantization and low-rank approximation do not address the KV cache's inherent linear growth with context length and quadratic complexity of attention, and thus remain insufficient for truly long-context inference.
“However, they do not address the cache's inherent linear growth with context length and quadratic complexity of attention, and thus remain insufficient for truly long-context inference.”
From IndexMem: Learned KV-Cache Eviction with Latent Memory for Long-Context LLM Inference (2026) · cites this paper as Liu et al., 2024
The Background (p. 3) gives a KV-cache shape proportional to prompt plus generated tokens, and the Introduction (p. 1) describes quantization as reducing the bytes occupied by that cache. These passages support the point that quantization does not remove growth with context length. Our copy does not establish the claim about low-rank approximation, quadratic attention complexity, or the conclusion that these approaches are insufficient for truly long-context inference. (LLM, conf=0.84)
Experimental — accuracy not yet measuredExisting remedies either quantize every cache entry, which is safe but bounded, or evict/merge tokens permanently, which can destroy information that becomes relevant only after the future query is known.
“Existing remedies either quantize every cache entry, which is safe but bounded, or evict/merge tokens permanently, which can destroy information that becomes relevant only after the future query is known.”
From Tamarin (HSQ): Reversible Hierarchical KV-Cache Compression for LLM Inference (2026) · cites this paper as Liu et al., 2024
The Introduction (p. 1) describes quantization and token eviction as approaches to reducing KV-cache size, and Related Work (p. 9) contrasts eviction with KIVI’s retention of all input tokens at lower precision. But the paper does not establish that quantization is uniformly safe, discuss token merging, or say that evicted information may become relevant only after a future query is known. The Introduction (p. 2) also says KIVI keeps a residual portion of the cache in full precision, rather than quantizing every entry. (LLM, conf=0.86)
Experimental — accuracy not yet measuredQuantization and low-rank approximation primarily reduce the per-token memory footprint of KV caches.
“Recent advances in KV cache compression, such as quantization (Liu et al., 2024; Hooper et al., 2024) and low-rank approximation (Chang et al., 2025), primarily reduce the per-token memory footprint.”
From IndexMem: Learned KV-Cache Eviction with Latent Memory for Long-Context LLM Inference (2026) · cites this paper as Liu et al., 2024
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
The publisher hasn’t deposited this paper’s reference list.
Research works printed but not linked: 6
- Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems , 33:1877-1901, 2020.
- Reiner Pope, Sholto Douglas, Aakanksha Chowdhery, Jacob Devlin, James Bradbury, Jonathan Heek, Kefan Xiao, Shivani Agrawal, and Jeff Dean. Efficiently scaling transformer inference. Proceedings of Machine Learning and Systems , 5, 2023.
- Ying Sheng, Lianmin Zheng, Binhang Yuan, Zhuohan Li, Max Ryabinin, Beidi Chen, Percy Liang, Christopher Ré, Ion Stoica, and Ce Zhang. Flexgen: High-throughput generative inference of large language models with a single gpu. In International Conference on Machine Learning , pages 31094-31116. PMLR, 2023.
- Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. Smoothquant: Accurate and efficient post-training quantization for large language models. In International Conference on Machine Learning , pages 38087-38099. PMLR, 2023a.
- Jiayi Yuan, Ruixiang Tang, Xiaoqian Jiang, and Xia Hu. Large language models for healthcare data augmentation: An example on patient-trial matching. In AMIA Annual Symposium Proceedings , volume 2023, page 1324. American Medical Informatics Association, 2023.
- Yilong Zhao, Chien-Yu Lin, Kan Zhu, Zihao Ye, Lequn Chen, Size Zheng, Luis Ceze, Arvind Krishnamurthy, Tianqi Chen, and Baris Kasikci. Atom: Low-bit quantization for efficient and accurate llm serving. Proceedings of Machine Learning and Systems , 6:196-209, 2024.
Cited by
Sorted by citations
SGLang: Efficient Execution of Structured Language Model Programs
Zheng, L., Yin, L., Xie, Z., Sun, C., Huang, J., Yu, C. H., Cao, S., Kozyrakis, C., Stoica, I., Gonzalez, J. E., et al. (2023) · arXiv · doi:10.48550/arxiv.2312.07104 · Cited by 145 · see citing works
HotPrefix: Hotness-Aware KV Cache Scheduling for Efficient Prefix Sharing in LLM Inference Systems
Li, Y., Gu, R., Huan, C., Wang, Z., Yao, R., Tian, C., & Chen, G. (2025) · Proceedings of the ACM on Management of Data · doi:10.1145/3749168 · Cited by 9 · see citing works
Stop Overthinking: A Survey on Efficient Reasoning for Large Language Models
Sui, Y., Chuang, Y.-N., Wang, G., Zhang, J., Zhang, T., Yuan, J., Liu, H., Wen, A., Zhong, S., Zou, N., et al. (2025) · arXiv · Cited by 2 · see citing works
Self-citationAn, Y., Lu, C., Zhu, K., Yu, T., Zhao, C., Wu, H., Tang, M., & Wang, J. (2026) · arXiv · doi:10.48550/arxiv.2605.08840
PolyKV: Shared Asymmetric TurboQuant-Compressed KV Memory for Multi-Agent LLMs
Patel, I., & Joshi, I. (2026) · Ishan Patel · doi:10.5281/zenodo.19686729
IndexMem: Learned KV-Cache Eviction with Latent Memory for Long-Context LLM Inference
Yang, X., Gu, H., Xu, B., Li, L., Liu, B., Liu, J., Zhu, Q., Guo, Y., & Han, S. (2026) · arXiv · doi:10.48550/arxiv.2605.25475
Make Each Token Count: Towards Improving Long-Context Performance with KV Cache Eviction
Bui, N., Nguyen, H. T., Cohan, A., & Ying, R. (2026) · arXiv · doi:10.48550/arxiv.2605.09649
Tamarin (HSQ): Reversible Hierarchical KV-Cache Compression for LLM Inference
Bazakov, A., & Kiselev, K. (2026) · doi:10.21203/rs.3.rs-10297225/v1
OjaKV: Context-Aware Online Low-Rank KV Cache Compression
Zhu, Y., Yang, D. H., Amiri, M. M., Murugesan, K., Pedapati, T., & Chen, P.-Y. (2025) · arXiv · doi:10.48550/arxiv.2509.21623
GEAR: An Efficient KV Cache Compression Recipe for Near-Lossless Generative Inference of LLM
Kang, H., Zhang, Q., Kundu, S., Jeong, G., Liu, Z., Krishna, T., & Zhao, T. (2024) · arXiv · doi:10.48550/arxiv.2403.05527 · Cited by 2 · see citing works
“When combining with KIVI, bu ff er size should be larger than group size.”
BackgroundMethodResultInfluentialSelf-citation
- Authors
- Liu, Z., Yuan, J., Jin, H., Zhong, S., Xu, Z., Braverman, V., Chen, B., & Hu, X.
- Affiliations
- Rice University (US)
- Texas A&M University (US)
- Stevens Institute of Technology (US)
- Carnegie Mellon University (US)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2024
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2402.02750
- OpenAlex
- W6966825443
- Cited by
- 0
- References
- 0
- Type
- posted content
- Language
- en
- Open access
- green
- doi.org
- Licence
- Locations
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Hardware and Architecture › Parallel Computing and Optimization Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- KV cache quantization, key-value cache, large language models, 2-bit quantization, batch size scaling, inference throughput
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
