GEAR: An Efficient KV Cache Compression Recipe for Near-Lossless Generative Inference of LLM
Open accessAbstract
Key-value (KV) caching has become the de-facto to accelerate generation speed for large language models (LLMs) inference. However, the growing cache demand with increasing sequence length has transformed LLM inference to be a memory bound problem, significantly constraining the system throughput. Existing methods rely on dropping unimportant tokens or quantizing all entries uniformly. Such methods, however, often incur high approximation errors to represent the compressed matrices. The autoregressive decoding process further compounds the error of each step, resulting in critical deviation in model generation and deterioration of performance. To tackle this challenge, we propose GEAR, an efficient KV cache compression framework that achieves near-lossless high-ratio compression. GEAR first applies quantization to majority of entries of similar magnitudes to ultra-low precision. It then employs a low rank matrix to approximate the quantization error, and a sparse matrix to remedy individual errors from outlier entries. By adeptly integrating three techniques, GEAR is able to fully exploit their synergistic potentials. Our experiments demonstrate that compared to alternatives, GEAR achieves near-lossless 4-bit KV cache compression with up to 2.38x throughput improvement, while reducing peak-memory size up to 2.29x. Our code is publicly available at https://github.com/HaoKang-Timmy/GEAR.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
SGLang is compatible with many other common inference optimizations.
“Additionally, SGLang is compatible with many other common inference optimizations [60, 39, 3, 23, 59, 10, 26, 15, 19, 32, 31, 11].”
From SGLang: Efficient Execution of Structured Language Model Programs (2023) · cites this paper as [19]
The Discussion and Conclusions section describes GEAR as a plug-and-play KV-cache compression method that improves inference throughput, so the paper supplies an example of an inference optimization. Our copy does not establish that SGLang is compatible with GEAR or with other optimizations; a search for “SGLang” found no relevant passage. (LLM, conf=0.79)
Experimental — accuracy not yet measuredThe KV cache causes memory usage to grow linearly with the number of tokens, and unmanaged growth can result in the GPU running out of memory.
“However, the KV cache causes memory usage to grow linearly with the number of tokens, and unmanaged growth can result in the GPU running out of memory [Hooper et al., 2024, Kang et al., 2024].”
From Online Scheduling for LLM Inference with KV Cache Constraints (2025) · cites this paper as Kang et al., 2024
Recipe-based cache compression such as GEAR targets a different bottleneck: near-lossless 4-bit caches buy capacity (longer contexts or more concurrent streams in fixed memory) at the price of additional decompression compute.
“Recipe-based cache compression such as GEAR [10] targets a different bottleneck: near-lossless 4-bit caches buy capacity (longer contexts or more concurrent streams in fixed memory) at the price of additional decompression compute.”
From Pre-Compiled Pipeline Shards for Distributed LLM Inference on Intel AI PC Fleets (2026) · cites this paper as [10]
System prompts are shared by all queries in various LLM applications and often exceed 1,000 tokens.
“For example, system prompts are shared by all queries in various LLM applications, such as Claude [4], ChatGPT [44] and Grok [58], and the promopts often exceed 1,000 tokens [28].”
From HotPrefix: Hotness-Aware KV Cache Scheduling for Efficient Prefix Sharing in LLM Inference Systems (2025) · cites this paper as [28]
The Background section discusses input tokens and KV caching, and Section 4.1 discusses chain-of-thought prompts, but neither addresses the length of system prompts. Searches for “system prompt,” “1,000 tokens,” and prompt length found no passage saying that system prompts often exceed 1,000 tokens; the cited paper’s copy therefore does not verify the part attached to [28]. (LLM, conf=0.76)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · cited 1× · see citing works
“Autoregressive large language models (LLMs) (Brown et al., 2020b; Zhang et al., 2022; Touvron et al., 2023a,b) have marked a significant milestone in natural language processing (NLP) and Value cache per-token, compressing the cache size by a high ratio.”
BackgroundPyTorch: An Imperative Style, High-Performance Deep Learning Library
Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., et al. (2019) · arXiv · doi:10.48550/arxiv.1912.01703 · Cited by 31,473 · cited 2× · see citing works
“We apply GEAR and baseline methods to open-source pre-trained LLMs available at Huggingface Wolf et al. (2019), using our inference framework written in PyTorch Paszke et al. (2019).”
MethodLanguage Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · cited 1× · see citing works
“Autoregressive large language models (LLMs) (Brown et al., 2020b; Zhang et al., 2022; Touvron et al., 2023a,b) have marked a significant milestone in natural language processing (NLP) and Value cache per-token, compressing the cache size by a high ratio.”
BackgroundQuantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference
Jacob, B., Kligys, S., Chen, B., Zhu, M.-L., Tang, M., Howard, A. G., Adam, H., & Kalenichenko, D. (2017) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.48550/arxiv.1712.05877 · Cited by 5,073 · cited 2×
“Group-wise quantization is widely applied to compress KV cacheSheng et al. (2023); Liu et al. (2024); Hooper et al. (2024).”
MethodPointer Sentinel Mixture ModelsPreprint
Merity, S., Xiong, C., Bradbury, J., & Socher, R. (2016) · International Conference on Learning Representations · doi:10.48550/arxiv.1609.07843 · Cited by 5,051 · cited 1× · see citing works
“We increase the batch size until out of memory and report the peak memory/throughput between FP16 KV caches and 2-bit quantization: KIVI, GEAR, and GEAR-L.”
MethodLLaMA: Open and Efficient Foundation Language Models
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. (2023) · arXiv · doi:10.48550/arxiv.2302.13971 · Cited by 4,602 · cited 4× · see citing works
“We use GEAR as a plug-and-play KV cache compression for generative inference with various LLM models (including LLaMA2-7B/13B Implementation optimization and details.”
BackgroundMethodInfluentialChain-Of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q. V., & Zhou, D. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1800 · Cited by 4,505 · see citing works
Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q., & Zhou, D. (2022) · arXiv · doi:10.48550/arxiv.2201.11903 · Cited by 4,263 · cited 1× · see citing works
“However, a stark contrast emerges when applying these methods to complex generative tasks that require models to generate longer responses or involve reasoning, such as mathematical problem-solving (Cobbe et al., 2021) and chain-of-thought (CoT) reasoning (Wei et al., 2023).”
BackgroundHuggingFace's Transformers: State-of-the-art Natural Language Processing
Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., et al. (2019) · arXiv · doi:10.48550/arxiv.1910.03771 · Cited by 3,347 · see citing works
Llama 2: Open Foundation and Fine-Tuned Chat Models
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2307.09288 · Cited by 2,604 · cited 3× · see citing works
“We use GEAR as a plug-and-play KV cache compression for generative inference with various LLM models (including LLaMA2-7B/13B Implementation optimization and details.”
BackgroundMethodBERT Rediscovers the Classical NLP Pipeline
Tenney, I., Das, D., & Pavlick, E. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1452 · Cited by 1,307 · see citing works
Emergent Abilities of Large Language Models
Wei, J., Tay, Y., Bommasani, R., Raffel, C., Zoph, B., Borgeaud, S., Yogatama, D., Bosma, M., Zhou, D., Metzler, D., et al. (2022) · arXiv · doi:10.48550/arxiv.2206.07682 · Cited by 1,264 · cited 1× · see citing works
“This phenomenon can be attributed to the non-trivial approximation error induced by them, i.e., di ff erence between original KV and the compressed ones.”
BackgroundFlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness
Dao, T., Fu, D. Y., Ermon, S., Rudra, A., & Ré, C. (2022) · arXiv · doi:10.48550/arxiv.2205.14135 · Cited by 1,214 · cited 1× · see citing works
“39 × compared to FP16 KV cache baseline.”
BackgroundAnalyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned
Voita, E., Talbot, D., Moiseev, F., Sennrich, R., & Titov, I. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1580 · Cited by 1,146 · see citing works
LaMDA: Language Models for Dialog Applications
Thoppilan, R., De Freitas, D., Hall, J., Shazeer, N., Kulshreshtha, A., Cheng, H.-T., Jin, A., Bos, T., Baker, L., Du, Y., et al. (2022) · arXiv · doi:10.48550/arxiv.2201.08239 · Cited by 778 · cited 1× · see citing works
“This phenomenon can be attributed to the non-trivial approximation error induced by them, i.e., di ff erence between original KV and the compressed ones.”
BackgroundTraining Verifiers to Solve Math Word Problems
Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., et al. (2021) · arXiv · doi:10.48550/arxiv.2110.14168 · Cited by 575 · cited 2× · see citing works
“However, a stark contrast emerges when applying these methods to complex generative tasks that require models to generate longer responses or involve reasoning, such as mathematical problem-solving (Cobbe et al., 2021) and chain-of-thought (CoT) reasoning (Wei et al., 2023).”
BackgroundPowerSGD: Practical Low-Rank Gradient Compression for Distributed OptimizationPreprint
Vogels, T., Karimireddy, S. P., & Jaggi, M. (2019) · Neural Information Processing Systems · doi:10.48550/arxiv.1905.13727 · Cited by 497 · see citing works
Zafrir, O., Boudoukh, G., Izsak, P., & Wasserblat, M. (2019) · 2019 Fifth Workshop on Energy Efficient Machine Learning and Cognitive Computing - NeurIPS Edition (EMC2-NIPS) · doi:10.1109/emc2-nips53020.2019.00016 · Cited by 442 · cited 1× · see citing works
“Such a composite approximation decouples the coherent parts from incoherent parts of the approximation error: the low-rank matrix captures the majority of coherent basis of quantization error while the sparse matrix rectifies the incoherency existing in individual outliers.”
BackgroundAtom: Low-bit Quantization for Efficient and Accurate LLM ServingPreprint
Zhao, Y.-L., Lin, C.-Y., Zhu, K., Ye, Z., Chen, L., Zheng, S.-Z., Ceze, L., Krishnamurthy, A., Chen, T.-Q., & Kasikci, B. (2023) · Conference on Machine Learning and Systems · doi:10.48550/arxiv.2310.19102 · Cited by 411 · see citing works
SqueezeLLM: Dense-and-Sparse QuantizationPreprint
Kim, S., Hooper, C., Gholami, A., Dong, Z., Li, X., Shen, S., Mahoney, M. W., & Keutzer, K. (2023) · International Conference on Machine Learning · doi:10.48550/arxiv.2306.07629 · Cited by 410 · cited 2× · see citing works
“…them in full precision with a sparse matrix S = Filter s ( X ) where Then, we perform the quantization on the extracted matrix and obtain the quantized backbone: The outlier extraction technique has been applied by Kim et al. (2023) to augment training-dependent non-uniform weight quantization.”
BackgroundMethodLongBench: A Bilingual, Multitask Benchmark for Long Context Understanding
Bai, Y., Lv, X., Zhang, J., Lyu, H., Tang, J., Huang, Z., Du, Z., Liu, X., Zeng, A., Hou, L., et al. (2024) · Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2024.acl-long.172 · Cited by 400 · see citing works
Wordcraft: Story Writing With Large Language Models
Yuan, A., Coenen, A., Reif, E., & Ippolito, D. (2022) · 27th International Conference on Intelligent User Interfaces · doi:10.1145/3490099.3511105 · Cited by 374 · cited 1× · see citing works
“This phenomenon can be attributed to the non-trivial approximation error induced by them, i.e., di ff erence between original KV and the compressed ones.”
BackgroundProgram Induction by Rationale Generation: Learning to Solve and Explain Algebraic Word Problems
Ling, W., Yogatama, D., Dyer, C., & Blunsom, P. (2017) · Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p17-1015 · Cited by 325 · cited 4× · see citing works
“To further demonstrate the e ff ective-ness of error reduction, we study the performance variation of GEAR-L when applying low-rank approximation to varying number of tokens with LLaMA3-8B on GSM8k and AQuA (w. CoT).”
BackgroundInfluentialLLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale
Dettmers, T., Lewis, M., Belkada, Y., & Zettlemoyer, L. (2022) · arXiv · doi:10.48550/arxiv.2208.07339 · Cited by 305 · see citing works
DeepSpeed- Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scale
Aminabadi, R. Y., Rajbhandari, S., Awan, A. A., Li, C., Li, D., Zheng, E., Ruwase, O., Smith, S., Zhang, M., Rasley, J., et al. (2022) · SC22: International Conference for High Performance Computing, Networking, Storage and Analysis · doi:10.1109/sc41404.2022.00051 · Cited by 297 · cited 2× · see citing works
“As an example, Figure 1 presents the approximation error of various methods on GSM8k and illustrates the deviation in token generations due to the accumulated error, which degrades the accuracy a lot.”
MethodMeasuring Massive Multitask Language Understanding
Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., & Steinhardt, J. (2020) · arXiv · doi:10.48550/arxiv.2009.03300 · Cited by 291 · cited 2× · see citing works
“Regarding the inference e ffi ciency, compared to the FP16 baseline, GEAR can reduce the peak memory up to 2.39 × , bring 2.10 × ∼ 5.07 × throughput improvement.”
BackgroundMethodInfluentialLoftQ: LoRA-Fine-Tuning-Aware Quantization for Large Language ModelsPreprint
Li, Y.-X., Yu, Y., Liang, C., He, P., Karampatziakis, N., Chen, W.-Z., & Zhao, T. (2023) · International Conference on Learning Representations · doi:10.48550/arxiv.2310.08659 · Cited by 267 · cited 1× · see citing works
“This idea has been introduced by Li et al. (2023) to optimize a similar objective for an accurate initialization of weight quantization.”
MethodSelf-citationChallenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them
Suzgun, M., Scales, N., Schärli, N., Gehrmann, S., Tay, Y., Chung, H. W., Chowdhery, A., Le, Q., Chi, E., Zhou, D., et al. (2023) · Findings of the Association for Computational Linguistics: ACL 2023 · doi:10.18653/v1/2023.findings-acl.824 · Cited by 253 · see citing works
SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models
Xiao, G., Lin, J., Seznec, M., Wu, H., Demouth, J., & Han, S. (2022) · arXiv · doi:10.48550/arxiv.2211.10438 · Cited by 247 · cited 1× · see citing works
“It is important to note that, compared to weights, KV cache compression presents unique challenges because KV caches can contain more outliers, making its accurate quantization more challenging than weights (Xiao et al., 2023).”
BackgroundGPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers
Frantar, E., Ashkboos, S., Hoefler, T., & Alistarh, D. (2022) · arXiv · doi:10.48550/arxiv.2210.17323 · Cited by 197 · see citing works
SparQ Attention: Bandwidth-Efficient LLM Inference
Ribar, L., Chelombiev, I., Hudlass-Galley, L., Blake, C., Luschi, C., & Orr, D. (2023) · International Conference on Machine Learning · doi:10.48550/arxiv.2312.04985 · Cited by 163 · cited 1×
“39 × compared to FP16 KV cache baseline.”
BackgroundZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformers
Yao, Z., Aminabadi, R. Y., Zhang, M., Wu, X., Li, C., & He, Y. (2022) · arXiv · doi:10.48550/arxiv.2206.01861 · Cited by 151 · cited 1× · see citing works
“For GEAR, we fix the sparsity ratio s at 2%, set the rank r to 4 for inputs in prefill phase, and set the rank to 2 for each group of n b new tokens in decoding phase.”
MethodReLU Strikes Back: Exploiting Activation Sparsity in Large Language Models
Mirzadeh, I., Alizadeh-Vahid, K., Mehta, S., del Mundo, C. C., Tuzel, O., Samei, G., Rastegari, M., & Farajtabar, M. (2023) · International Conference on Learning Representations · doi:10.48550/arxiv.2310.04564 · Cited by 145 · cited 1×
“39 × compared to FP16 KV cache baseline.”
BackgroundEfficiently Scaling Transformer Inference
Pope, R., Douglas, S., Chowdhery, A., Devlin, J., Bradbury, J., Levskaya, A., Heek, J., Xiao, K., Agrawal, S., & Dean, J. (2022) · arXiv · doi:10.48550/arxiv.2211.05102 · Cited by 113 · cited 1× · see citing works
“Consequently, a relatively large approximation error does not significantly hinder the generation of target tokens.”
BackgroundFlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPU
Sheng, Y., Zheng, L., Yuan, B., Li, Z., Ryabinin, M., Fu, D. Y., Xie, Z., Chen, B., Barrett, C., Gonzalez, J. E., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.06865 · Cited by 86 · cited 1× · see citing works
“We compare GEAR with the following baseline methods: • Per-token group-wise quantization (used in FlexGen (Sheng et al., 2023)) is a widely-adopted method that quantizes KV cache per-token with fine-grained grouping.”
MethodKVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization
Hooper, C., Kim, S., Mohammadzadeh, H., Mahoney, M. W., Shao, Y. S., Keutzer, K., & Gholami, A. (2024) · arXiv · doi:10.48550/arxiv.2401.18079 · Cited by 69 · cited 2× · see citing works
“Group-wise quantization is widely applied to compress KV cacheSheng et al. (2023); Liu et al. (2024); Hooper et al. (2024).”
MethodLiu, Z., Desai, A., Liao, F., Wang, W., Xie, V., Xu, Z., Kyrillidis, A., & Shrivastava, A. (2023) · Advances in Neural Information Processing Systems 36 · doi:10.52202/075280-2279 · Cited by 50 · see citing works
Self-citationLiu, Z., Desai, A., Liao, F., Wang, W., Xie, V., Xu, Z., Kyrillidis, A., & Shrivastava, A. (2023) · arXiv · doi:10.48550/arxiv.2305.17118 · Cited by 50 · cited 1×
“Generally speaking, our framework consists of three components to decompose KV matrices: (i) First, we apply an existing quantization method to e ffi ciently quantize the majority (e.g., 98”
MethodSelf-citationAdaLoRA: Adaptive Budget Allocation for Parameter-Efficient Fine-Tuning
Zhang, Q., Chen, M., Bukharin, A., Karampatziakis, N., He, P., Cheng, Y., Chen, W., & Zhao, T. (2023) · arXiv · doi:10.48550/arxiv.2303.10512 · Cited by 47 · cited 3× · see citing works
“• H 2 O (Zhang et al., 2023b) is a recent token dropping method evicting unimportant tokens with lower accumulated attention scores, which we compare with in Table 10.”
BackgroundMethodResultSelf-citationChallenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them
Suzgun, M., Scales, N., Schärli, N., Gehrmann, S., Tay, Y., Chung, H. W., Chowdhery, A., Le, Q. V., Chi, E. H., Zhou, D., et al. (2022) · arXiv · doi:10.48550/arxiv.2210.09261 · Cited by 44 · cited 3× · see citing works
“With the CoT demonstrations, we have the average prefill length of GSM8k, AQuA, and BBH as 900, 1304, 1021 respectively (see Appendix 10).”
BackgroundModel Tells You What to Discard: Adaptive KV Cache Compression for LLMs
Ge, S., Zhang, Y., Liu, L., Zhang, M., Han, J., & Gao, J. (2023) · arXiv · doi:10.48550/arxiv.2310.01801 · Cited by 13 · cited 1× · see citing works
“Their performance deteriorates under a high compression ratio * (e.g., 4-bit/2-bit quantization or dropping > 50% tokens (Ge et al., 2023)), which is noticeable in both types of methods * .”
BackgroundLongBench: A Bilingual, Multitask Benchmark for Long Context Understanding
Bai, Y., Lv, X., Zhang, J., Lyu, H., Tang, J., Huang, Z., Du, Z., Liu, X., Zeng, A., Hou, L., et al. (2023) · arXiv · Cited by 9 · cited 4× · see citing works
“Specifically, we evaluate the performance with LLaMA2-7B on LongBench (Bai et al., 2023), which is a suit of 21 long-context understanding tasks including question answering, summarization, code completion, etc. (please see Appendix 10 for task metrics and dataset statistics).”
MethodInfluentialKIVI : Plug-and-play 2bit KV Cache Quantization with Streaming Asymmetric Quantization
Liu, Z., Yuan, J., Jin, H., Zhong, S., Xu, Z., Braverman, V., Chen, B., & Hu, X. (2023) · Unpublished · doi:10.13140/rg.2.2.28167.37282 · Cited by 6 · cited 17× · see citing works
“When combining with KIVI, bu ff er size should be larger than group size.”
BackgroundMethodResultInfluentialSelf-citationKIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache
Liu, Z., Yuan, J., Jin, H., Zhong, S., Xu, Z., Braverman, V., Chen, B., & Hu, X. (2024) · arXiv · doi:10.48550/arxiv.2402.02750 · cited 17×
“When combining with KIVI, bu ff er size should be larger than group size.”
BackgroundMethodResultInfluentialSelf-citationTell Your Model Where to Attend: Post-hoc Attention Steering for LLMs
Zhang, Q., Singh, C., Liu, L., Liu, X., Yu, B., Gao, J., & Zhao, T. (2023) · arXiv · doi:10.48550/arxiv.2311.02262
Self-citationOPT: Open Pre-trained Transformer Language Models
Zhang, S., Roller, S., Goyal, N., Artetxe, M., Chen, M., Chen, S., Dewan, C., Diab, M., Li, X., Lin, X. V., et al. (2022) · arXiv · doi:10.48550/arxiv.2205.01068 · cited 1×
“Autoregressive large language models (LLMs) (Brown et al., 2020b; Zhang et al., 2022; Touvron et al., 2023a,b) have marked a significant milestone in natural language processing (NLP) and Value cache per-token, compressing the cache size by a high ratio.”
BackgroundChain-of-Thought Hub: A Continuous Effort to Measure Large Language Models' Reasoning Performance
Fu, Y., Ou, L., Chen, M., Wan, Y., Peng, H., & Khot, T. (2023) · arXiv · doi:10.48550/arxiv.2305.17306 · cited 1×
“Given the complexity of these tasks, we use the chain-of-thought prompts created by (Fu et al., 2023) to improve reasoning, which contains 8-shot demonstrations of multi-step reasoning.”
Method
Flagged first, then by citations
Research works printed but not linked: 1
- Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A. et al. (2020a). Language models are few-shot learners. Advances in neural information processing systems, 33 1877-1901.
Cited by
Sorted by citations
LLM Inference Serving: Survey of Recent Advances and Opportunities
Li, B., Jiang, Y., Gadepally, V. N., & Tiwari, D. (2024) · IEEE High Performance Extreme Computing Conference · doi:10.1109/hpec62836.2024.10938426 · Cited by 42 · see citing works
“Gear [23] achieves near-lossless high-ratio KV cache compression by quantizing the majority of entries of similar magnitudes and employs a low-rank matrix to approximate the quantization error.”
SGLang: Efficient Execution of Structured Language Model Programs
Zheng, L., Yin, L., Xie, Z., Sun, C., Huang, J., Yu, C. H., Cao, S., Kozyrakis, C., Stoica, I., Gonzalez, J. E., et al. (2023) · arXiv · doi:10.48550/arxiv.2312.07104 · Cited by 145 · see citing works
“Additionally, SGLang is compatible with many other common inference optimizations [60, 39, 3, 23, 59, 10, 26, 15, 19, 32, 31, 11].”
HotPrefix: Hotness-Aware KV Cache Scheduling for Efficient Prefix Sharing in LLM Inference Systems
Li, Y., Gu, R., Huan, C., Wang, Z., Yao, R., Tian, C., & Chen, G. (2025) · Proceedings of the ACM on Management of Data · doi:10.1145/3749168 · Cited by 9 · see citing works
“Quantization methods [11, 22, 29, 37, 44, 48, 62, 69], aim to compress KV caches by reducing their bit precision, from 16-bit to as low as 4-bit or even lower.”
Mitigating Hallucinations in Vision-Language Models through Image-Guided Head Suppression
Sarkar, S., Che, Y., Gavin, A., Beerel, P. A., & Kundu, S. (2025) · Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2025.emnlp-main.631 · Cited by 4 · see citing works
Self-citationOnline Scheduling for LLM Inference with KV Cache Constraints
Jaillet, P., Jiang, J., Mellou, K., Molinaro, M., Podimata, C., & Zhou, Z. (2025) · arXiv · doi:10.48550/arxiv.2502.07115 · Cited by 1 · see citing works
“However, the KV cache causes memory usage to grow linearly with the number of tokens, and unmanaged growth can result in the GPU running out of memory [Hooper et al., 2024, Kang et al., 2024].”
Cross-layer Attention Sharing for Pre-trained Large Language Models
Mu, Y., Wu, Y., Fan, Y., Wang, C. Q., Li, H., Zeng, J., He, Q., Yang, M., Meng, F., Zhou, J., et al. (2026) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl.a.616 · Cited by 9 · see citing works
“One line of work along this research is reducing the KV cache by cutting down useless to-kens (Liu et al., 2023a; Zhang et al., 2023; Xiao et al., 2024) or compressing the representation of KV cache (DeepSeek-AI et al., 2024; Kang et al., 2024).”
Pre-Compiled Pipeline Shards for Distributed LLM Inference on Intel AI PC Fleets
Berenbaum, T., & Venkatachalam, M. (2026) · arXiv · doi:10.48550/arxiv.2608.19147
“GEAR [10] compresses cache entries to 4 bits near-losslessly by pairing uniform quantization with a low-rank approximation of the residual and a sparse correction for outliers; SkipKV [16] skips KV generation and storage outright for semantically redundant spans of long reasoning traces.”
FlexPrefill: A Context-Aware Sparse Attention Mechanism for Efficient Long-Sequence Inference
Lai, X., Lu, J., Luo, Y., Ma, Y., & Zhou, X. (2025) · arXiv · doi:10.48550/arxiv.2502.20766
“Other approaches aim to reduce KV cache sizes through quantization (Liu et al., 2024a; Kang et al., 2024; Liu et al., 2024b), token merging (Nawrot et al., 2024; Li et al., 2024), or KV discarding (Zhang et al., 2023; Dong et al., 2024).”
TailorKV: A Hybrid Framework for Long-Context Inference via Tailored KV Cache Optimization
Yao, D., Shen, B., Lin, Z., Liu, W., Luan, J., Wang, B., & Wang, W. (2025) · arXiv · doi:10.48550/arxiv.2505.19586 · Cited by 2
“For example, KVQuant (Hooper et al., 2024) iso-lates outliers for mixed precision, GEAR (Kang et al., 2024) utilizes SVD to recover residuals, and KIVI (Liu et al., 2024d) quantizes keys per channel and values per token.”
- Authors
- Kang, H., Zhang, Q., Kundu, S., Jeong, G., Liu, Z., Krishna, T., & Zhao, T.
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2024
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2403.05527
- OpenAlex
- W4392678657
- Cited by
- 2
- References
- 43
- Citations by year
- 2026: 1 · 2024: 1
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- key-value cache compression, large language model inference, low-rank approximation, sparse matrices, quantization, autoregressive decoding, inference throughput, memory reduction, 4-bit quantization
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
