Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs
Open accessAbstract
In this study, we introduce adaptive KV cache compression, a plug-and-play method that reduces the memory footprint of generative inference for Large Language Models (LLMs). Different from the conventional KV cache that retains key and value vectors for all context tokens, we conduct targeted profiling to discern the intrinsic structure of attention modules. Based on the recognized structure, we then construct the KV cache in an adaptive manner: evicting long-range contexts on attention heads emphasizing local contexts, discarding non-special tokens on attention heads centered on special tokens, and only employing the standard KV cache for attention heads that broadly attend to all tokens. Moreover, with the lightweight attention profiling used to guide the construction of the adaptive KV cache, FastGen can be deployed without resource-intensive fine-tuning or re-training. In our experiments across various asks, FastGen demonstrates substantial reduction on GPU memory consumption with negligible generation quality loss. We will release our code and the compatible CUDA kernel for reproducibility.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Existing context-compression methods often use distillation to produce an offline prompt, memory, or compressed KV representation before generation.
“Existing context-compression methods often use distillation to produce an offline prompt, memory, or compressed KV representation before generation [7, 28, 53, 32, 19, 34, 56, 30, 31].”
From Learning What to Remember: Test-Time Training via Context Distillation (2026) · cites this paper as [19]
The Introduction (page unavailable) says FastGen compresses the KV cache “on the fly,” and Section 3.2 says it profiles the prompt before managing the cache during token generation. The Introduction also says it requires no fine-tuning. This is contrary to describing this cited method as using distillation to produce an offline compressed representation before generation. (LLM, conf=0.88)
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe large number of prefix KV caches and limited GPU memory make it impractical to store all prefix KV caches in GPU memory.
“Nevertheless, the large number of prefix KV caches and the limited GPU memory capacity make it impractical to store all prefix KV caches in GPU memory.”
From HotPrefix: Hotness-Aware KV Cache Scheduling for Efficient Prefix Sharing in LLM Inference Systems (2025) · cites this paper as [17]
The Introduction (page not provided) says KV-cache memory use grows rapidly and can exceed GPU capacity, requiring offloading. It does not specifically discuss storing a large number of distinct prefix KV caches, so that part of the claim is not established by this paper. (LLM, conf=0.85)
Experimental — accuracy not yet measuredExisting methods have not effectively addressed the challenges encountered in real-world applications, where prompts are exceptionally long yet require accurate information retrieval.
“In summary, existing methods have not effectively addressed the challenges encountered in realworld applications, where prompts are exceptionally long yet require accurate information retrieval.”
From SnapKV: LLM Knows What You are Looking for Before Generation (2024) · cites this paper as [8]
Section 2 (Related Work; page unavailable) identifies limitations of earlier compression methods, and Section 3.2 says previous KV-cache approaches do not account for differing attention structures. The paper does not frame that shortcoming specifically as a failure to retrieve accurate information from exceptionally long prompts in real-world applications. (LLM, conf=0.82)
Experimental — accuracy not yet measuredThe important attention features in LLMs change with different instructions, challenging static compression methods.
“This variability challenges the effectiveness of static compression methods that depend on constant weighted importance or fixed policies [7, 6, 8].”
From SnapKV: LLM Knows What You are Looking for Before Generation (2024) · cites this paper as [8]
Sections 3.2 and 4.1 say attention structures differ across heads and layers, making a uniform KV-cache policy suboptimal. Section 3.3 selects policies using each encoded prompt, but the cited passages do not establish that important attention features change with different instructions specifically. The source supports the challenge to fixed policies, not that stated explanation for the variability. (LLM, conf=0.83)
Experimental — accuracy not yet measuredStreamingLLM and FastGen keep only a fixed window plus a few important tokens for long-sequence serving.
“StreamingLLM [33] and FastGen [6] keep only a fixed window plus a few important tokens for long-sequence serving.”
From TokenDance: Scaling Multi-Agent LLM Serving via Collective KV Cache Sharing (2026) · cites this paper as [6]
Section 3.4 (page unavailable) describes FastGen policies that can retain local context and selected special or frequently attended tokens, including a hybrid that combines them. But Section 3.2 says FastGen selects a compression strategy for each attention head; it does not uniformly keep a fixed window plus a few important tokens. This verdict concerns FastGen [6], not StreamingLLM [33]. (LLM, conf=0.88)
Experimental — accuracy not yet measured
GPT-3 requires approximately 4.5 MB of KV cache per token.
“For example, GPT-3 requires approximately 4.5 MB of KV cache per token [17].”
From HotPrefix: Hotness-Aware KV Cache Scheduling for Efficient Prefix Sharing in LLM Inference Systems (2025) · cites this paper as [17]
The Introduction and Section 3.1 explain that KV-cache memory grows as tokens are generated, but neither gives a per-token figure for GPT-3. Searches for “GPT-3 requires approximately 4.5 MB” and “4.5 MB GPT-3 cache token” found no passage supporting the claimed amount. Page numbers were not available in the indexed copy. (LLM, conf=0.74)
Experimental — accuracy not yet measuredCachedAttention extends KV cache storage across disk, host memory, and GPU memory, mainly for multi-turn conversation scenarios.
“CachedAttention [17] extends KV cache storage across disk, host memory, and GPU memory, primarily targeting multi-turn conversation scenarios.”
From HotPrefix: Hotness-Aware KV Cache Scheduling for Efficient Prefix Sharing in LLM Inference Systems (2025) · cites this paper as [17]
The Introduction discusses KV-cache offloading to CPU/NVMe, but does not identify CachedAttention or describe storage across disk, host memory, and GPU memory. Searches for CachedAttention and multi-turn conversation found no passage addressing the claimed system or its target scenario. (LLM, conf=0.76)
Experimental — accuracy not yet measuredExisting KV cache compression methods mainly focus on compressing the KV cache appended during decoding steps, while overlooking compression of prompt KV cache.
“they mainly focus on compressing the KV cache appended during decoding steps, while overlooking the realistic problem of compressing KV cache for prompts, which is typically the bottleneck in memory efficiency.”
From SnapKV: LLM Knows What You are Looking for Before Generation (2024) · cites this paper as [8]
Section 3.2 distinguishes prompt encoding from token generation in FastGen, but does not say that existing methods mainly compress KV cache appended during decoding. Section 2 (Related Work) even describes an earlier method that compresses prompts. The passages do not establish that prompt KV cache is typically the memory-efficiency bottleneck. (LLM, conf=0.72)
Experimental — accuracy not yet measuredMost existing KV cache eviction methods lack detailed evaluation in long-context settings.
“However, most of these methods lack a detailed evaluation in long-context settings.”
From SnapKV: LLM Knows What You are Looking for Before Generation (2024) · cites this paper as [8]
The Related Work section describes earlier KV-cache methods but does not assess whether they were evaluated in long-context settings. Section 5.1 describes this paper’s own benchmarks and comparisons, not the claimed lack of detailed evaluation in most existing methods. Searches for long-context evaluation and prior methods found no passage addressing that claim. (LLM, conf=0.75)
Experimental — accuracy not yet measuredFastGen faces the similar problem with H2O.
“Nonetheless, it faces the similar problem with H2O.”
From SnapKV: LLM Knows What You are Looking for Before Generation (2024) · cites this paper as [8]
Section 5.1 identifies H2O as a fixed KV-cache compression baseline, while Section 3.2 describes FastGen’s adaptive approach. Neither passage identifies a problem shared by the two methods, and the citing sentence does not specify what problem it means. Searches for H2O-related limitations and shared eviction problems did not settle the claim. (LLM, conf=0.71)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
The publisher hasn’t deposited this paper’s reference list.
Research works printed but not linked: 22
- Reza Yazdani Aminabadi, Samyam Rajbhandari, Minjia Zhang, Ammar Ahmad Awan, Cheng Li, Du Li, Elton Zheng, Jeff Rasley, Shaden Smith, Olatunji Ruwase, and Yuxiong He. Deepspeed- inference: Enabling efficient inference of transformer models at unprecedented scale. SC22: International Conference for High Performance Computing, Networking, Storage and Analysis, pp. 1-15, 2022.
- Sebastian Bach, Alexander Binder, Grégoire Montavon, Frederick Klauschen, Klaus-Robert Müller, and Wojciech Samek. On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation. PLoS ONE, 10, 2015. URL https://api.semanticscholar. org/CorpusID:9327892.
- Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In Hugo Larochelle, Marc'Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin (eds.), Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Pro- cessing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020.
- Víctor Campos, Brendan Jou, Xavier Giró i Nieto, Jordi Torres, and Shih-Fu Chang. Skip rnn: Learning to skip state updates in recurrent neural networks. ArXiv, abs/1708.06834, 2017. URL https://api.semanticscholar.org/CorpusID:1859294. · arXiv
- Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021. · arXiv
- Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. Generating long sequences with sparse transformers. CoRR, abs/1904.10509, 2019. URL http://arxiv.org/abs/1904.10509. · arXiv
- Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021. · arXiv
- Zihang Dai, Guokun Lai, Yiming Yang, and Quoc Le. Funnel-transformer: Filter- ing out sequential redundancy for efficient language processing. In Hugo Larochelle, Marc'Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin (eds.), Advances in Neural Information Processing Systems 33: Annual Conference on Neu- ral Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, vir- tual, 2020. URL https://proceedings.neurips.cc/paper/2020/hash/ 2cd2915e69546904e4e5d4a2ac9e1652-Abstract.html.
- Saurabh Goyal, Anamitra R. Choudhury, Saurabh Raje, Venkatesan T. Chakaravarthy, Yogish Sab- harwal, and Ashish Verma. Power-bert: Accelerating bert inference via progressive word- vector elimination. In International Conference on Machine Learning, 2020. URL https: //api.semanticscholar.org/CorpusID:219792793.
- Christian Hansen, Casper Hansen, Stephen Alstrup, Jakob Grue Simonsen, and Christina Lioma. Neural speed reading with structural-jump-lstm. ArXiv, abs/1904.00761, 2019. URL https: //api.semanticscholar.org/CorpusID:90258012. · arXiv
- Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. arXiv preprint arXiv:1904.09751, 2019. · arXiv
- Aniruddha Kembhavi, Minjoon Seo, Dustin Schwenk, Jonghyun Choi, Ali Farhadi, and Han- naneh Hajishirzi. Are you smarter than a sixth grader? textbook question answering for multi- modal machine comprehension. 2017 IEEE Conference on Computer Vision and Pattern Recog- nition (CVPR), pp. 5376-5384, 2017. URL https://api.semanticscholar.org/ CorpusID:1310550.
- Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Al- berti, Danielle Epstein, Illia Polosukhin, Matthew Kelcey, Jacob Devlin, Kenton Lee, Kristina N. Toutanova, Llion Jones, Ming-Wei Chang, Andrew Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. Natural questions: a benchmark for question answering research. Transactions of the Association of Computational Linguistics, 2019.
- Zichang Liu, Jue Wang, Tri Dao, Tianyi Zhou, Binhang Yuan, Zhao Song, Anshumali Shrivas- tava, Ce Zhang, Yuandong Tian, Christopher Ré, and Beidi Chen. Deja vu: Contextual spar- sity for efficient llms at inference time. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), International Confer- ence on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pp. 22137-22176. PMLR, 2023b. URL https://proceedings.mlr.press/v202/liu23am.html.
- Paul Michel, Omer Levy, and Graham Neubig. Are sixteen heads really better than one? In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 32.
- Curran Associates, Inc., 2019. URL https://proceedings.neurips.cc/paper_files/paper/2019/ file/2c601ad9d2ff9bc8b282670cdd54f69f-Paper.pdf.
- Minjoon Seo, Sewon Min, Ali Farhadi, and Hannaneh Hajishirzi. Neural speed reading via skim-rnn. ArXiv, abs/1711.02085, 2017. URL https://api.semanticscholar.org/ CorpusID:3140413. · arXiv
- Noam M. Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc V. Le, Geoffrey E. Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of- experts layer. ArXiv, abs/1701.06538, 2017. URL https://api.semanticscholar. org/CorpusID:12462234. · arXiv
- Ying Sheng, Lianmin Zheng, Binhang Yuan, Zhuohan Li, Max Ryabinin, Daniel Y. Fu, Zhiqiang Xie, Beidi Chen, Clark W. Barrett, Joseph Gonzalez, Percy Liang, Christopher Ré, Ioan Cristian Stoica, and Ce Zhang. High-throughput generative inference of large language models with a single gpu. In International Conference on Machine Learning, 2023. URL https://api. semanticscholar.org/CorpusID:257495837.
- Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023a. · arXiv
- Elena Voita, David Talbot, Fedor Moiseev, Rico Sennrich, and Ivan Titov. Analyzing multi-head self-attention: Specialized heads do the heavy lifting, the rest can be pruned, July 2019. URL https://aclanthology.org/P19-1580.
- Wangchunshu Zhou, Canwen Xu, Tao Ge, Julian J. McAuley, Ke Xu, and Furu Wei. BERT loses patience: Fast and robust inference with early exit. In Hugo Larochelle, Marc'Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin (eds.), Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020. URL https://proceedings.neurips.cc/paper/2020/hash/ d4dd111a4fd973394238aca5c05bebe3-Abstract.html.
Cited by
Sorted by citations
VideoLLM-online: Online Video Large Language Model for Streaming Video
Chen, J., Lv, Z., Wu, S., Lin, K. Q., Song, C., Gao, D., Liu, J.-W., Gao, Z., Mao, D., & Shou, M. Z. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01742 · Cited by 83 · see citing works
InfLLM: Training-Free Long-Context Extrapolation for LLMs with an Efficient Context Memory
Xiao, C., Zhang, P., Han, X., Xiao, G., Lin, Y., Zhang, Z., Liu, Z., & Sun, M. (2024) · Advances in Neural Information Processing Systems 37 · doi:10.52202/079017-3801 · Cited by 14 · see citing works
SGLang: Efficient Execution of Structured Language Model Programs
Zheng, L., Yin, L., Xie, Z., Sun, C., Huang, J., Yu, C. H., Cao, S., Kozyrakis, C., Stoica, I., Gonzalez, J. E., et al. (2023) · arXiv · doi:10.48550/arxiv.2312.07104 · Cited by 145 · see citing works
“Additionally, SGLang is compatible with many other common inference optimizations [60, 39, 3, 23, 59, 10, 26, 15, 19, 32, 31, 11].”
HotPrefix: Hotness-Aware KV Cache Scheduling for Efficient Prefix Sharing in LLM Inference Systems
Li, Y., Gu, R., Huan, C., Wang, Z., Yao, R., Tian, C., & Chen, G. (2025) · Proceedings of the ACM on Management of Data · doi:10.1145/3749168 · Cited by 9 · see citing works
TightLLM: Maximizing Throughput for LLM Inference via Adaptive Offloading Policy
Hu, Y., Liu, X., Yang, G., Li, L., Zeng, K., Zhao, Z., Chen, S., Zhao, L., Li, W., & Li, K. (2025) · IEEE Transactions on Computers · doi:10.1109/tc.2025.3558009 · Cited by 8 · see citing works
Enabling Efficient Batch Serving for LMaaS via Generation Length Prediction
Cheng, K., Hu, W., Wang, Z., Du, P., Li, J., & Zhang, S. (2024) · IEEE International Conference on Web Services (ICWS) · doi:10.1109/icws62655.2024.00104 · Cited by 9 · see citing works
Enabling Efficient Batch Serving for LMaaS via Generation Length Prediction
Cheng, K., Hu, W., Wang, Z., Du, P., Li, J., & Zhang, S. (2024) · arXiv · doi:10.48550/arxiv.2406.04785 · Cited by 7 · see citing works
SnapKV: LLM Knows What You are Looking for Before Generation
Li, Y., Huang, Y., Yang, B., Venkitesh, B., Locatelli, A., Ye, H., Cai, T., Lewis, P., & Chen, D. (2024) · arXiv · doi:10.48550/arxiv.2404.14469 · Cited by 74 · see citing works
“There are many perspectives to mitigate these problems, including KV cache eviction during token generation [5–8].”
BackgroundMethodAn Empirical Study on the Power Consumption of LLMs with Different GPU Platforms
Chen, Z., Lin, W., Xie, X., Hu, Y., Li, C., Tong, Q., Wu, Y., & Li, S. (2024) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata62323.2024.10825662 · Cited by 6 · see citing works
Anchor Attention, Small Cache: Code Generation With Large Language Models
Zhang, X., Zhou, Y., Yang, G., Gall, H. C., & Chen, T. (2025) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2025.3570680 · Cited by 6 · see citing works
FINCH: Prompt-guided Key-Value Cache Compression for Large Language Models
Corallo, G., & Papotti, P. (2024) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00716 · Cited by 5 · see citing works
ChunkKV: Semantic-Preserving KV Cache Compression for Efficient Long-Context LLM Inference
Liu, X., Tang, Z., Dong, P., Li, Z., Liu, Y., Li, B., Hu, X., & Chu, X. (2025) · arXiv · doi:10.48550/arxiv.2502.00299 · Cited by 4 · see citing works
“By retaining coherent chunks of text, it provides language models with more contextually rich and semantically complete information, leading to improved performance in tasks that require deep understanding and accurate information retrieval from extensive documents.”
End-to-End Acceleration of Generative Models With Runtime Regularized KV Cache Management
Moradifirouzabadi, A., & Kang, M. (2025) · IEEE Journal on Emerging and Selected Topics in Circuits and Systems · doi:10.1109/jetcas.2025.3568716 · Cited by 4 · see citing works
Key, Value, Compress: A Systematic Exploration of KV Cache Compression Techniques
Javidnia, N., Rouhani, B. D., & Koushanfar, F. (2025) · Custom Integrated Circuits Conference · doi:10.1109/cicc63670.2025.10983416 · Cited by 4 · see citing works
MoEcho: Exploiting Side-Channel Attacks to Compromise User Privacy in Mixture-of-Experts LLMs
Ding, R., Xu, T., Shen, X., Ding, A. A., & Fei, Y. (2025) · ACM SIGSAC Conference on Computer and Communications Security (CCS) · doi:10.1145/3719027.3765174 · Cited by 4 · see citing works
PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funneling
Cai, Z., Zhang, Y., Gao, B., Liu, Y., Li, Y., Liu, T., Lu, K., Xiong, W., Dong, Y., Hu, J., et al. (2024) · arXiv · doi:10.48550/arxiv.2406.02069 · Cited by 2 · see citing works
“To tackle these memory constraints, recent studies have explored the optimization of KV caching, including approaches such as low-rank decomposition of the KV cache [8] or pruning non-essential KV cache [34, 22, 12].”
BackgroundSelf-citationFrom Human Memory to AI Memory: A Survey on Memory Mechanisms in the Era of LLMs
Wu, Y., Liang, S., Zhang, C., Wang, Y., Zhang, Y., Guo, H., Tang, R., & Liu, Y. (2025) · arXiv · doi:10.48550/arxiv.2504.15965 · Cited by 3 · see citing works
“LookupFFN [109], ChunkKV [110], vLLM [111], FastServe [112], StreamingLLM [113], Orca [114], DistServe [115], LLM.int8() [116], FastGen [117], Train Large, Then Compress [118], Scissorhands [119], , Mooncake [121], MemServe [122], SLM Serving [123], IMPRESS [124], AdaServe [125], MPIC [126],…”
Self-citationsLLM: Accelerating LLM Inference using Semantic Load Balancing with Shared Memory Data Structures
Lin, J., Zhang, S. Q., & Leon‐Garcia, A. (2024) · International Symposium on Quality Electronic Design (ISQED) · doi:10.1109/isqed60706.2024.10528703 · Cited by 2 · see citing works
Keep the Cost Down: A Review on Methods to Optimize LLM' s KV-Cache Consumption
Shi, L., Zhang, H., Yao, Y., Li, Z., & Zhao, H. (2024) · arXiv · doi:10.48550/arxiv.2407.18003 · Cited by 2
GEAR: An Efficient KV Cache Compression Recipe for Near-Lossless Generative Inference of LLM
Kang, H., Zhang, Q., Kundu, S., Jeong, G., Liu, Z., Krishna, T., & Zhao, T. (2024) · arXiv · doi:10.48550/arxiv.2403.05527 · Cited by 2 · see citing works
“Their performance deteriorates under a high compression ratio * (e.g., 4-bit/2-bit quantization or dropping > 50% tokens (Ge et al., 2023)), which is noticeable in both types of methods * .”
BackgroundAn, Y., Lu, C., Zhu, K., Yu, T., Zhao, C., Wu, H., Tang, M., & Wang, J. (2026) · arXiv · doi:10.48550/arxiv.2605.08840
“…attention redistribution, ReST-KV considers its impact to improve KV retention. methods typically rely on fixed attention patterns (Han et al., 2024; Ge et al., 2023) or use statistical information from attention weights (Zhang et al., 2023; Li et al., 2024b; Cai et al., 2024) to estimate the…”
TokenDance: Scaling Multi-Agent LLM Serving via Collective KV Cache Sharing
Bian, Z., Wu, F., Zhang, C., Dong, H., Liang, Y., & Zhuo, Y. (2026) · arXiv · doi:10.48550/arxiv.2604.03143
Learning What to Remember: Test-Time Training via Context Distillation
Wang, Z., Dang, X., Zhu, R.-J., Wen, Z., Fu, H., Chai, W., & Lee, J. D. (2026) · arXiv · doi:10.48550/arxiv.2608.01672
“Existing context-compression methods often use distillation to produce an offline prompt, memory, or compressed KV representation before generation [7, 28, 53, 32, 19, 34, 56, 30, 31].”
A Survey of Agent Memory in the Second Half: Towards Self-Evolving and Long-Horizon Agents
Huang, W.-C., Zhang, W., Liang, Y., Bei, Y., Chen, Y., Feng, T., Pan, X., Tan, Z., Wang, Y., Wei, T., et al. (2026) · arXiv · doi:10.48550/arxiv.2602.06052
Self-citationDon't Break the Cache: An Evaluation of Prompt Caching for Long-Horizon Agentic Tasks
Lumer, E., Nizar, F., Jangiti, A., Frank, K., Gulati, A., Phadate, M., & Subbiah, V. K. (2026) · arXiv · doi:10.48550/arxiv.2601.06007
CritiPrefill: A Segment-wise Criticality-based Approach for Prefilling Acceleration in LLMs
Lv, J., Feng, Y., Xie, X., Jia, X., Peng, Q., & Xie, G. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10887916
Wen, Z., Xu, L., & Wang, M. (2025) · Integrated Circuits and Systems · doi:10.23919/ics.2025.3575371
Corm: Cache Optimization with Recent Message for Large Language Model InferencePreprint
Huang, Z., Jiang, H., & Dai, J. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5388897
Cartridges: Lightweight and general-purpose long context representations via self-study
Eyuboglu, S., Ehrlich, R., Arora, S., Guha, N., Zinsley, D., Liu, E., Tennien, W., Rudra, A., Zou, J., Mirhoseini, A., et al. (2025) · arXiv · doi:10.48550/arxiv.2506.06266
- Authors
- Ge, S., Zhang, Y., Liu, L., Zhang, M., Han, J., & Gao, J.
- Affiliations
- University of Illinois Urbana-Champaign (US)
- Microsoft (United States) (US)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2023
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2310.01801
- OpenAlex
- W4387355950
- Cited by
- 13
- References
- 0
- Citations by year
- 2025: 8 · 2024: 5
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Funders
- National Science Foundation (2019897, 1956151, 2118329, HR001121C0165)
- Defense Advanced Research Projects Agency (FA8750-19-2-1004, FA8750, HR001121C0165)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Keywords
- KV cache compression, large language models, memory footprint reduction, generative inference, special tokens, GPU memory consumption, CUDA kernels
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
