PyramidInfer: Pyramid KV Cache Compression for High-throughput LLM Inference
Open accessAbstract
Large Language Models (LLMs) have shown remarkable comprehension abilities but face challenges in GPU memory usage during inference, hindering their scalability for real-time applications like chatbots. To accelerate inference, we store computed keys and values (KV cache) in the GPU memory. Existing methods study the KV cache compression to reduce memory by pruning the pre-computed KV cache. However, they neglect the inter-layer dependency between layers and huge memory consumption in pre-computation. To explore these deficiencies, we find that the number of crucial keys and values that influence future generations decreases layer by layer and we can extract them by the consistency in attention weights. Based on the findings, we propose PyramidInfer, a method that compresses the KV cache by layer-wise retaining crucial context. PyramidInfer saves significant memory by computing fewer keys and values without sacrificing performance. Experimental results show PyramidInfer improves 2.2x throughput compared to Accelerate with over 54% GPU memory reduction in KV cache.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
PyramidInfer discards tokens in earlier layers, preventing them from being reconsidered in later layers.
“PyramidInfer discards tokens in earlier layers, preventing them from being reconsidered in later layers.”
From PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funneling (2024) · cites this paper as Yang et al., 2024
Section 4.1 (Method; page unavailable) says PyramidInfer retains selected keys and values at each layer, including shallow layers, so it does discard some cache entries there. It also says the method retains more context in shallow layers and selects entries layer by layer; it does not establish that an entry discarded in an earlier layer cannot be reconsidered in a later one. (LLM, conf=0.83)
Experimental — accuracy not yet measuredPyramidInfer employs a geometric decay strategy, while PyramidKV adopts an arithmetic decay strategy.
“While PyramidInfer Yang et al. (2024) employs a geometric decay strategy, our method adopts an arithmetic decay strategy.”
From PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funneling (2024) · cites this paper as Yang et al., 2024
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
No references could be resolved yet.
Research works printed but not linked: 2
- Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. 2021. Evaluating large language models trained on code.
- Suyu Ge, Yunan Zhang, Liyuan Liu, Minjia Zhang, Jiawei Han, and Jianfeng Gao. 2023. Model tells you what to discard: Adaptive kv cache compression for llms. arXiv preprint arXiv:2310.01801. · arXiv
Cited by
Sorted by citations
HotPrefix: Hotness-Aware KV Cache Scheduling for Efficient Prefix Sharing in LLM Inference Systems
Li, Y., Gu, R., Huan, C., Wang, Z., Yao, R., Tian, C., & Chen, G. (2025) · Proceedings of the ACM on Management of Data · doi:10.1145/3749168 · Cited by 9 · see citing works
ChunkKV: Semantic-Preserving KV Cache Compression for Efficient Long-Context LLM Inference
Liu, X., Tang, Z., Dong, P., Li, Z., Liu, Y., Li, B., Hu, X., & Chu, X. (2025) · arXiv · doi:10.48550/arxiv.2502.00299 · Cited by 4 · see citing works
“Layer-WisePolicy Semantic Information Efficient Index Reuse StreamingLLM [8] ✓ H2O [11] ✓ ✓ SnapKV [15] ✓ ✓ PyramidInfer [19] ✓ ✓ ✓ PyramidKV [17] ✓ ✓ ✓ In light of this, we observe that complete semantic information usually appears in a continuous sequence [4, 22–24].”
PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funneling
Cai, Z., Zhang, Y., Gao, B., Liu, Y., Li, Y., Liu, T., Lu, K., Xiong, W., Dong, Y., Hu, J., et al. (2024) · arXiv · doi:10.48550/arxiv.2406.02069 · Cited by 2 · see citing works
Keep the Cost Down: A Review on Methods to Optimize LLM' s KV-Cache Consumption
Shi, L., Zhang, H., Yao, Y., Li, Z., & Zhao, H. (2024) · arXiv · doi:10.48550/arxiv.2407.18003 · Cited by 2
Self-citationPolyKV: Shared Asymmetric TurboQuant-Compressed KV Memory for Multi-Agent LLMs
Patel, I., & Joshi, I. (2026) · Ishan Patel · doi:10.5281/zenodo.19686729
An, Y., Lu, C., Zhu, K., Yu, T., Zhao, C., Wu, H., Tang, M., & Wang, J. (2026) · arXiv · doi:10.48550/arxiv.2605.08840
- Authors
- Yang, D., Han, X., Gao, Y., Hu, Y., Zhang, S., & Zhao, H.
- Affiliations
- Shanghai Jiao Tong University (CN)
- Xiaohongshu (Little Red Book) (China) (CN)
- South China University of Technology (CN)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2024
- Pages
- 3258-3270
- arXiv
- 2405.12532
- OpenAlex
- W4402670433
- Cited by
- 31
- References
- 0
- Citations by year
- 2026: 20 · 2025: 10 · 2024: 1
- FWCI
- 9.16
- Type
- conference paper
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- Findings of the Association for Computational Linguistics: ACL (Findings of ACL) (publishedVersion)
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Keywords
- KV cache compression, large language model inference, GPU memory reduction, attention weights, inference throughput
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
