CoChat
Sign Up
PreprintCited by 7Parallel Computing and Optimization TechniquesTopic ModelingNatural Language Processing Techniques

The DOI matches its DataCite record, with a small difference in title or year; its reference list is not available.

DiffKV: Differentiated Memory Management for Large Language Models with Parallel KV Compaction

Open access
Zhang, Yanqi, Hu, Yuwei, Zhao, Runyuan, Lui, John C. S., Chen, Haibo·2024·arXiv
DOI 10.48550/arxiv.2412.03131arXiv 2412.03131
Open this paper in CoChat — free account

Abstract

Large language models (LLMs) demonstrate remarkable capabilities but face substantial serving costs due to their high memory demands, with the key-value (KV) cache being a primary bottleneck. State-of-the-art KV cache compression techniques, such as quantization and pruning, apply uniform treatment to both keys and values, and discard unimportant tokens entirely, overlooking the fine-grained distinctions in the significance of individual KV cache components. To address such limitations, we introduce \textit{DiffKV}, a novel framework for efficient KV cache compression that exploits three levels of differentiation in the KV cache: (1) the differing impact of keys and values on attention computation, (2) the varying importance of tokens, and (3) the diverse dynamic sparsity patterns across attention heads. These levels of differentiation introduce irregular memory usage patterns across different requests and attention heads, posing significant scalability challenges for memory management. To address these challenges, DiffKV proposes an on-GPU memory manager that compacts fragmented free memory list into contiguous regions in parallel, effectively translating sparsity in the KV cache into performance gains. We evaluate DiffKV on several mainstream LLMs, including the emerging thinking models that generate extended chains of thought. DiffKV is able to compress the KV cache by $2.7\times$ to $5.7\times$ with near-lossless accuracy on complex workloads requiring sophisticated reasoning and long-generation capabilities, and enhances throughput by $1.9\times$ to $5.4\times$. Source codes of DiffKV are available at https://github.com/zyqCSL/DiffKV.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2412.03131

Cites

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

No citation lists for this paper

No citation lists for this paper yet — checked again daily.

References 20 · Cited by 7

Research works printed but not linked: 72

  1. [4][4] ACHIAM, J., ADLER, S., AGARWAL, S., AHMAD, L., AKKAYA, I., ALEMAN, F. L., ALMEIDA, D., ALTENSCHMIDT, J., ALTMAN, S., ANADKAT, S., ET AL. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 (2023). - · arXiv
  2. [5][5] ADNAN, M., ARUNKUMAR, A., JAIN, G., NAIR, P. J., SOLOVEYCHIK, I., AND KAMATH, P. Keyformer: Kv cache reduction through key tokens selection for efficient generative inference. Proceedings of Machine Learning and Systems 6 (2024), 114-127. -
  3. [6][6] AINSLIE, J., LEE-THORP, J., DE JONG, M., ZEMLYANSKIY, Y., LEBRÓN, F., AND SANGHAI, S. Gqa: Training generalized multi-query transformer models from multi-head checkpoints. arXiv preprint arXiv:2305.13245 (2023). - · arXiv
  4. [7][7] AMINABADI, R. Y., RAJBHANDARI, S., AWAN, A. A., LI, C., LI, D., ZHENG, E., RUWASE, O., SMITH, S., ZHANG, M., RASLEY, J., ET AL. Deepspeed-inference: enabling efficient inference of transformer models at unprecedented scale. In SC22: International Conference for High Performance Computing, Networking, Storage and Analysis (2022), IEEE, pp. 1-15. -
  5. [8][8] AUSTIN, J., ODENA, A., NYE, M., BOSMA, M., MICHALEWSKI, H., DOHAN, D., JIANG, E., CAI, C., TERRY, M., LE, Q., ET AL. Program synthesis with large language models. arXiv preprint arXiv:2108.07732 (2021). - · arXiv
  6. [9][9] BAI, Y., LV, X., ZHANG, J., LYU, H., TANG, J., HUANG, Z., DU, Z., LIU, X., ZENG, A., HOU, L., ET AL. Longbench: A bilingual, multitask benchmark for long context understanding. arXiv preprint arXiv:2308.14508 (2023). - · arXiv
  7. [10][10] BROWN, T. B. Language models are few-shot learners. arXiv preprint arXiv:2005.14165 (2020). - · arXiv
  8. [11][11] CAI, Z., ZHANG, Y., GAO, B., LIU, T., LU, K., XIONG, W., DONG, Y., CHANG, B., HU, J., AND XIAO, W. Pyramidkv: Dynamic kv cache compression based on pyramidal information funneling. arXiv preprint arXiv:2406.02069 (2024). - · arXiv
  9. [12][12] CHEN, M., TWOREK, J., JUN, H., YUAN, Q., PINTO, H. P. D. O., KAPLAN, J., EDWARDS, H., BURDA, Y., JOSEPH, N., BROCKMAN, G., ET AL. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374 (2021). - · arXiv
  10. [14][14] CHIANG, W.-L., ZHENG, L., SHENG, Y., ANGELOPOULOS, A. N., LI, T., LI, D., ZHANG, H., ZHU, B., JORDAN, M., GONZALEZ, J. E., ET AL. Chatbot arena: An open platform for evaluating llms by human preference. arXiv preprint arXiv:2403.04132 (2024). - · arXiv
  11. [15][15] COBBE, K., KOSARAJU, V., BAVARIAN, M., CHEN, M., JUN, H., KAISER, L., PLAPPERT, M., TWOREK, J., HILTON, J., NAKANO, R., ET AL. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168 (2021). - · arXiv
  12. [16][16] DAO, T. Flashattention-2: Faster attention with better parallelism and work partitioning. arXiv preprint arXiv:2307.08691 (2023). - · arXiv
  13. [17][17] DAO, T., FU, D., ERMON, S., RUDRA, A., AND RÉ, C. Flashattention: Fast and memory-efficient exact attention with io-awareness. Advances in Neural Information Processing Systems 35 (2022), 16344-16359. -
  14. [18][18] DETTMERS, T., LEWIS, M., BELKADA, Y., AND ZETTLEMOYER, L. Gpt3. int8 (): 8-bit matrix multiplication for transformers at scale. Advances in Neural Information Processing Systems 35 (2022), 30318-30332. -
  15. [19][19] DONG, S., CHENG, W., QIN, J., AND WANG, W. Qaq: Quality adaptive quantization for llm kv cache. arXiv preprint arXiv:2403.04643 (2024). - · arXiv
  16. [20][20] DUBEY, A., JAUHRI, A., PANDEY, A., KADIAN, A., AL-DAHLE, A., LETMAN, A., MATHUR, A., SCHELTEN, A., YANG, A., FAN, A., ET AL. The llama 3 herd of models. arXiv preprint arXiv:2407.21783 (2024). - · arXiv
  17. [21][21] FANG, J., YU, Y., ZHAO, C., AND ZHOU, J. Turbotransformers: an efficient gpu serving system for transformer models. In Proceedings of the 26th ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming (2021), pp. 389-402. -
  18. [22][22] GAO, P., YU, L., WU, Y., AND LI, J. Low latency rnn inference with cellular batching. In Proceedings of the Thirteenth EuroSys Conference (2018), pp. 1-15. -
  19. [23][23] GE, S., ZHANG, Y., LIU, L., ZHANG, M., HAN, J., AND GAO, J. Model tells you what to discard: Adaptive kv cache compression for llms. arXiv preprint arXiv:2310.01801 (2023). - · arXiv
  20. [25][25] GUO, C., TANG, J., HU, W., LENG, J., ZHANG, C., YANG, F., LIU, Y., GUO, M., AND ZHU, Y. Olive: Accelerating large language models via hardware-friendly outlier-victim pair quantization. In Proceedings of the 26. 50th Annual International Symposium on Computer Architecture (2023), pp. 1-15. -
  21. [26][26] GUO, D., YANG, D., ZHANG, H., SONG, J., ZHANG, R., XU, R., ZHU, Q., MA, S., WANG, P., BI, X., ET AL. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948 (2025). - · arXiv
  22. [27][27] GUO, D., ZHU, Q., YANG, D., XIE, Z., DONG, K., ZHANG, W., CHEN, G., BI, X., WU, Y., LI, Y., ET AL. Deepseek-coder: When the large language model meets programming-the rise of code intelligence. arXiv preprint arXiv:2401.14196 (2024). - · arXiv
  23. [28][28] HAN, M., ZHANG, H., CHEN, R., AND CHEN, H. Microsecond-scale preemption for concurrent { GPUaccelerated }{ DNN } inferences. In 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22) (2022), pp. 539-558. -
  24. [29][29] HARRIS, M., SENGUPTA, S., AND OWENS, J. D. Parallel prefix sum (scan) with cuda. GPU gems 3 , 39 (2007), 851-876. -
  25. [30][30] HE, Y., ZHANG, L., WU, W., LIU, J., ZHOU, H., AND ZHUANG, B. Zipcache: Accurate and efficient kv cache quantization with salient token identification. Advances in Neural Information Processing Systems 37 (2024), 68287-68307. -
  26. [31][31] HENDRYCKS, D., BURNS, C., BASART, S., ZOU, A., MAZEIKA, M., SONG, D., AND STEINHARDT, J. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300 (2020). - · arXiv
  27. [32][32] HOOPER, C., KIM, S., MOHAMMADZADEH, H., MAHONEY, M. W., SHAO, Y. S., KEUTZER, K., AND GHOLAMI, A. Kvquant: Towards 10 million context length llm inference with kv cache quantization. arXiv preprint arXiv:2401.18079 (2024). - · arXiv
  28. [33][33] HUANG, Y., CHENG, Y., BAPNA, A., FIRAT, O., CHEN, D., CHEN, M., LEE, H., NGIAM, J., LE, Q. V., WU, Y., ET AL. Gpipe: Efficient training of giant neural networks using pipeline parallelism. Advances in neural information processing systems 32 (2019). -
  29. [34][34] HUI, B., YANG, J., CUI, Z., YANG, J., LIU, D., ZHANG, L., LIU, T., ZHANG, J., YU, B., LU, K., ET AL. Qwen2. 5-coder technical report. arXiv preprint arXiv:2409.12186 (2024). - · arXiv
  30. [35][35] JAECH, A., KALAI, A., LERER, A., RICHARDSON, A., EL-KISHKY, A., LOW, A., HELYAR, A., MADRY, A., BEUTEL, A., CARNEY, A., ET AL. Openai o1 system card. arXiv preprint arXiv:2412.16720 (2024). - · arXiv
  31. [36][36] JIANG, A. Q., SABLAYROLLES, A., ROUX, A., MENSCH, A., SAVARY, B., BAMFORD, C., CHAPLOT, D. S., CASAS, D. D. L., HANNA, E. B., BRESSAND, F., ET AL. Mixtral of experts. arXiv preprint arXiv:2401.04088 (2024). - · arXiv
  32. [37][37] JOUPPI, N. P., YOUNG, C., PATIL, N., PATTERSON, D., AGRAWAL, G., BAJWA, R., BATES, S., BHATIA, S., BODEN, N., BORCHERS, A., ET AL. In-datacenter performance analysis of a tensor processing unit. In Proceedings of the 44th annual international symposium on computer architecture (2017), pp. 1-12. -
  33. [38][38] KAO, S.-C., SUBRAMANIAN, S., AGRAWAL, G., YAZDANBAKHSH, A., AND KRISHNA, T. Flat: An optimized dataflow for mitigating attention bottlenecks. In Proceedings of the 28th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2 (2023), pp. 295-310. -
  34. [39][39] KWON, W., LI, Z., ZHUANG, S., SHENG, Y., ZHENG, L., YU, C. H., GONZALEZ, J., ZHANG, H., AND STOICA, I. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Symposium on Operating Systems Principles (2023), pp. 611-626. -
  35. [40][40] LEE, W., LEE, J., SEO, J., AND SIM, J. { InfiniGen } : Efficient generative inference of large language models with dynamic { KV } cache management. In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24) (2024), pp. 155-172. -
  36. [41][41] LEWKOWYCZ, A., ANDREASSEN, A., DOHAN, D., DYER, E., MICHALEWSKI, H., RAMASESH, V., SLONE, A., ANIL, C., SCHLAG, I., GUTMAN-SOLO, T., ET AL. Solving quantitative reasoning problems with language models. Advances in Neural Information Processing Systems 35 (2022), 3843-3857. -
  37. [42][42] LI, Y., HUANG, Y., YANG, B., VENKITESH, B., LOCATELLI, A., YE, H., CAI, T., LEWIS, P., AND CHEN, D. Snapkv: Llm knows what you are looking for before generation. arXiv preprint arXiv:2404.14469 (2024). - · arXiv
  38. [43][43] LI, Z., ZHENG, L., ZHONG, Y., LIU, V., SHENG, Y., JIN, X., HUANG, Y., CHEN, Z., ZHANG, H., GONZALEZ, J. E., ET AL. { AlpaServe } : Statistical multiplexing with model parallelism for deep learning serving. In 17th USENIX Symposium on Operating Systems Design and Implementation (OSDI 23) (2023), pp. 663-679. -
  39. [44][44] LIN, Y., TANG, H., YANG, S., ZHANG, Z., XIAO, G., GAN, C., AND HAN, S. Qserve: W4a8kv4 quantization and system co-design for efficient llm serving. arXiv preprint arXiv:2405.04532 (2024). - · arXiv
  40. [45][45] LIU, A., FENG, B., WANG, B., WANG, B., LIU, B., ZHAO, C., DENGR, C., RUAN, C., DAI, D., GUO, D., ET AL. Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model. arXiv preprint arXiv:2405.04434 (2024). - · arXiv
  41. [46][46] LIU, J., XIA, C. S., WANG, Y., AND ZHANG, L. Is your code generated by chatGPT really correct? rigorous evaluation of large language models for code generation. In Thirty-seventh Conference on Neural Information Processing Systems (2023). -
  42. [47][47] LIU, Z., DESAI, A., LIAO, F., WANG, W., XIE, V., XU, Z., KYRILLIDIS, A., AND SHRIVASTAVA, A. Scissorhands: Exploiting the persistence of importance hypothesis for llm kv cache compression at test time. Advances in Neural Information Processing Systems 36 (2024). -
  43. [48][48] LIU, Z., YUAN, J., JIN, H., ZHONG, S., XU, Z., BRAVERMAN, V., CHEN, B., AND HU, X. Kivi: A tuning-free asymmetric 2bit quantization for kv cache. arXiv preprint arXiv:2402.02750 (2024). - · arXiv
  44. [49][49] MERITY, S., XIONG, C., BRADBURY, J., AND SOCHER, R. Pointer sentinel mixture models. arXiv preprint arXiv:1609.07843 (2016). - · arXiv
  45. [50][50] NAGEL, M., FOURNARAKIS, M., AMJAD, R. A., BONDARENKO, Y., VAN BAALEN, M., AND BLANKEVOORT, T. A white paper on neural network quantization. arXiv preprint arXiv:2106.08295 (2021). - · arXiv
  46. [51][51] NAKANO, K. An optimal parallel prefix-sums algorithm on the memory machine models for gpus. In International Conference on Algorithms and Architectures for Parallel Processing (2012), Springer, pp. 99-113. -
  47. [52][52] NARAYANAN, D., HARLAP, A., PHANISHAYEE, A., SESHADRI, V., DEVANUR, N. R., GANGER, G. R., GIBBONS, P. B., AND ZAHARIA, M. Pipedream: Generalized pipeline parallelism for dnn training. In Proceedings of the 27th ACM symposium on operating systems principles (2019), pp. 1-15. -
  48. [53][53] NARAYANAN, D., SHOEYBI, M., CASPER, J., LEGRESLEY, P., PATWARY, M., KORTHIKANTI, V., VAINBRAND, D., KASHINKUNTI, P., BERNAUER, J., CATANZARO, B., ET AL. Efficient large-scale language model training on gpu clusters using megatron-lm. In Proceedings of the international conference for high performance computing, networking, storage and analysis (2021), pp. 1-15. -
  49. [55][55] OUYANG, L., WU, J., JIANG, X., ALMEIDA, D., WAINWRIGHT, C., MISHKIN, P., ZHANG, C., AGARWAL, S., SLAMA, K., RAY, A., ET AL. Training language models to follow instructions with human feedback. Advances in neural information processing systems 35 (2022), 27730-27744. -
  50. [56][56] POPE, R., DOUGLAS, S., CHOWDHERY, A., DEVLIN, J., BRADBURY, J., HEEK, J., XIAO, K., AGRAWAL, S., AND DEAN, J. Efficiently scaling transformer inference. Proceedings of Machine Learning and Systems 5 (2023), 606-624. -
  51. [57][57] REIN, D., HOU, B. L., STICKLAND, A. C., PETTY, J., PANG, R. Y., DIRANI, J., MICHAEL, J., AND BOWMAN, S. R. Gpqa: A graduate-level google-proof q&a benchmark. In First Conference on Language Modeling (2024). -
  52. [58][58] SAAB, K., TU, T., WENG, W.-H., TANNO, R., STUTZ, D., WULCZYN, E., ZHANG, F., STROTHER, T., PARK, C., VEDADI, E., ET AL. Capabilities of gemini models in medicine. arXiv preprint arXiv:2404.18416 (2024). - · arXiv
  53. [59][59] SHAO, Z., WANG, P., ZHU, Q., XU, R., SONG, J., ZHANG, M., LI, Y., WU, Y., AND GUO, D. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300 (2024). - · arXiv
  54. [60][60] SHAZEER, N. Fast transformer decoding: One writehead is all you need. arXiv preprint arXiv:1911.02150 (2019). - · arXiv
  55. [61][61] SHENG, Y., ZHENG, L., YUAN, B., LI, Z., RYABININ, M., CHEN, B., LIANG, P., RÉ, C., STOICA, I., AND ZHANG, C. Flexgen: High-throughput generative inference of large language models with a single gpu. In International Conference on Machine Learning (2023), PMLR, pp. 31094-31116. -
  56. [62][62] SHI, Y., YANG, Z., XUE, J., MA, L., XIA, Y., MIAO, Z., GUO, Y., YANG, F., AND ZHOU, L. Welder: Scheduling deep learning memory access via tile-graph. In 17th USENIX Symposium on Operating Systems Design and Implementation (OSDI 23) (2023), pp. 701718. -
  57. [63][63] SHOEYBI, M., PATWARY, M., PURI, R., LEGRESLEY, P., CASPER, J., AND CATANZARO, B. Megatron-lm: Training multi-billion parameter language models using model parallelism. arXiv preprint arXiv:1909.08053 (2019). - · arXiv
  58. [64][64] TAGHANAKI, S. A., KHANI, A., AND KHASAHMADI, A. Mmlu-pro+: Evaluating higher-order reasoning and shortcut learning in llms. arXiv preprint arXiv:2409.02257 (2024). - · arXiv
  59. [65][65] TANG, J., ZHAO, Y., ZHU, K., XIAO, G., KASIKCI, B., AND HAN, S. Quest: Query-aware sparsity for efficient long-context llm inference. arXiv preprint arXiv:2406.10774 (2024). - · arXiv
  60. [66][66] TEAM, G., ANIL, R., BORGEAUD, S., WU, Y., ALAYRAC, J.-B., YU, J., SORICUT, R., SCHALKWYK, J., DAI, A. M., HAUTH, A., ET AL. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805 (2023). - · arXiv
  61. [68][68] TOUVRON, H., MARTIN, L., STONE, K., ALBERT, P., ALMAHAIRI, A., BABAEI, Y., BASHLYKOV, N., BATRA, S., BHARGAVA, P., BHOSALE, S., ET AL. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 (2023). - · arXiv
  62. [69][69] VASWANI, A. Attention is all you need. Advances in Neural Information Processing Systems (2017). -
  63. [70][70] XIAO, G., TANG, J., ZUO, J., GUO, J., YANG, S., TANG, H., FU, Y., AND HAN, S. Duoattention: Efficient long-context llm inference with retrieval and streaming heads. arXiv preprint arXiv:2410.10819 (2024). - · arXiv
  64. [71][71] XIAO, G., TIAN, Y., CHEN, B., HAN, S., AND LEWIS, M. Efficient streaming language models with attention sinks. arXiv preprint arXiv:2309.17453 (2023). - · arXiv
  65. [72][72] XIE, Y., WU, J., TU, H., YANG, S., ZHAO, B., ZONG, Y., JIN, Q., XIE, C., AND ZHOU, Y. A preliminary study of o1 in medicine: Are we closer to an ai doctor? arXiv preprint arXiv:2409.15277 (2024). - · arXiv
  66. [73][73] YANG, Y., ZHAO, L., LI, Y., ZHANG, H., LI, J., ZHAO, M., CHEN, X., AND LI, K. Infless: a native serverless system for low-latency, high-throughput inference. In Proceedings of the 27th ACM International Conference on Architectural Support for Programming Languages and Operating Systems (2022), pp. 768-781. -
  67. [74][74] YU, G.-I., JEONG, J. S., KIM, G.-W., KIM, S., AND CHUN, B.-G. Orca: A distributed serving system for { Transformer-Based } generative models. In 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22) (2022), pp. 521-538. -
  68. [75][75] ZHANG, H., TANG, Y., KHANDELWAL, A., AND STOICA, I. { SHEPHERD } : Serving { DNNs } in the wild. In 20th USENIX Symposium on Networked Systems Design and Implementation (NSDI 23) (2023), pp. 787-808. -
  69. [76][76] ZHANG, Z., SHENG, Y., ZHOU, T., CHEN, T., ZHENG, L., CAI, R., SONG, Z., TIAN, Y., RÉ, C., BARRETT, C., ET AL. H2o: Heavy-hitter oracle for efficient generative inference of large language models. Advances in Neural Information Processing Systems 36 (2024). -
  70. [77][77] ZHAO, Y., LIN, C.-Y., ZHU, K., YE, Z., CHEN, L., ZHENG, S., CEZE, L., KRISHNAMURTHY, A., CHEN, T., AND KASIKCI, B. Atom: Low-bit quantization for efficient and accurate llm serving. Proceedings of Machine Learning and Systems 6 (2024), 196-209. -
  71. [78][78] ZHONG, T., LIU, Z., PAN, Y., ZHANG, Y., ZHOU, Y., LIANG, S., WU, Z., LYU, Y., SHU, P., YU, X., ET AL. Evaluation of openai o1: Opportunities and challenges of agi. arXiv preprint arXiv:2409.18486 (2024). - · arXiv
  72. [79][79] ZUO, P., LIN, H., DENG, J., ZOU, N., YANG, X., DIAO, Y., GAO, W., XU, K., CHEN, Z., LU, S., ET AL. Serving large language models on huawei cloudmatrix384. arXiv preprint arXiv:2506.12708 (2025). · arXiv

Metadata

Authors
Zhang, Y., Hu, Y., Zhao, R., Lui, J. C. S., & Chen, H.
Affiliations
  • Huawei Technologies (China) (CN)
  • Chinese University of Hong Kong (HK)
  • Shanghai Jiao Tong University (CN)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2024
Pages
431-445
DOI
10.48550/arxiv.2412.03131
arXiv
2412.03131
OpenAlex
W4414735889
Cited by
7
References
20
Citations by year
2026: 7
FWCI
5.98
Type
conference paper
Language
en
Open access
Licence
Locations
  • ACM SIGOPS Symposium on Operating Systems Principles (SOSP) (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Hardware and Architecture › Parallel Computing and Optimization Techniques
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
Keywords
key-value cache compression, large language models, memory management, attention heads, token importance, large language model inference, inference throughput

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: DiffKV: Differentiated Memory Management for Large Language Models with Parallel KV Compaction. https://app.cochat.ai/papers/10.48550/arxiv.2412.03131
The paper
Zhang, Y., Hu, Y., Zhao, R., Lui, J. C. S., & Chen, H. (2024). DiffKV: Differentiated Memory Management for Large Language Models with Parallel KV Compaction. arXiv. https://doi.org/10.48550/arxiv.2412.03131