CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusion
Open accessAbstract
Large language models (LLMs) often incorporate multiple text chunks in their inputs to provide the necessary contexts. To speed up the prefill of the long LLM inputs, one can pre -compute the KV cache of a text and re -use the KV cache when the context is reused as the prefix of another LLM input. However, the reused text chunks are not always the input prefix, which makes precomputed KV caches not directly usable since they ignore the text’s cross-attention with the preceding texts. Thus, the benefits of reusing KV caches remain largely unrealized. This paper tackles just one challenge: when an LLM input contains multiple text chunks, how to quickly combine their precomputed KV caches in order to achieve the same generation quality as the expensive full prefill ( i.e., without reusing KV cache)? This challenge naturally arises in retrieval-augmented generation (RAG) where the input is supplemented with multiple retrieved texts as the context. We present CacheBlend , a scheme that reuses the pre-computed KV caches, regardless prefix or not, and selectively recomputes the KV values of a small subset of tokens to partially update each reused KV cache. In the meantime, the small extra delay for recomputing some tokens can be pipelined with the retrieval of KV caches within the same job, allowing CacheBlend to store KV caches in slower devices with more storage capacity while retrieving them without increasing the inference delay. By comparing CacheBlend with the state-of-the-art…
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
CacheBlend and EPIC are better suited to more regular contiguous reuse structures.
“CacheBlend and EPIC [7] are better suited to more regular contiguous reuse structures.”
From SparseX: Efficient Segment-Level KV Cache Sharing for Interleaved LLM Serving (2026) · cites this paper as [6]
CacheBlend targets RAG scenarios, concatenates multiple cached knowledge chunks into a new request, and selectively recomputes KV Cache tensors for a subset of tokens to restore cross-chunk interactions.
“For example, CacheBlend [6] targets RAG scenarios, concatenates multiple cached knowledge chunks into a new request, and selectively recomputes KV Cache tensors for a subset of tokens to restore cross-chunk interactions.”
From SparseX: Efficient Segment-Level KV Cache Sharing for Interleaved LLM Serving (2026) · cites this paper as [6]
Cites
MapReduce: simplified data processing on large clusters
Dean, J., & Ghemawat, S. (2008) · Communications of the ACM · doi:10.1145/1327452.1327492 · Cited by 18,476 · see citing works
Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks
Reimers, N., & Gurevych, I. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1410 · Cited by 12,778 · see citing works
Transformers: State-of-the-Art Natural Language Processing
Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., et al. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations · doi:10.18653/v1/2020.emnlp-demos.6 · Cited by 8,741 · see citing works
RoFormer: Enhanced transformer with rotary position embedding
Su, J., Ahmed, M., Lu, Y., Pan, S., Bo, W., & Liu, Y. (2024) · Neurocomputing · doi:10.1016/j.neucom.2023.127063 · Cited by 1,957 · see citing works
Self-citationEfficient Memory Management for Large Language Model Serving with PagedAttention
Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez, J., Zhang, H., & Stoica, I. (2023) · Proceedings of the 29th Symposium on Operating Systems Principles · doi:10.1145/3600006.3613165 · Cited by 1,817 · cited 5× · see citing works
“Several systems have followed this approach, e.g., vLLM [36], SGLang [59], and RAGCache [33].”
InfluentialEfficient Memory Management for Large Language Model Serving with PagedAttention
Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez, J. E., Zhang, H., & Stoica, I. (2023) · arXiv · doi:10.48550/arxiv.2309.06180 · Cited by 1,802 · see citing works
Rethinking Attention with Performers
Choromanski, K., Likhosherstov, V., Dohan, D., Song, X., Gane, A., Sarlos, T., Hawkins, P., Davis, J., Mohiuddin, A., Kaiser, L., et al. (2020) · arXiv · doi:10.48550/arxiv.2009.14794 · Cited by 695 · cited 1× · see citing works
“If we recompute 𝑟 % of tokens per layer, the total compute overhead will be 𝑟 % of full prefill.”
In-Context Retrieval-Augmented Language Models
Ram, O., Levine, Y., Dalmedigos, I., Muhlgay, D., Shashua, A., Leyton-Brown, K., & Shoham, Y. (2023) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00605 · Cited by 449 · cited 1× · see citing works
“Retrieval augmented generation (RAG): RAG [22, 23, 37, 46, 48] can enhance the accuracy and reliability of LLMs with text chunks fetched from external sources.”
SAMSum Corpus: A Human-annotated Dialogue Dataset for Abstractive Summarization
Gliwa, B., Mochol, I., Biesek, M., & Wawer, A. (2019) · Proceedings of the 2nd Workshop on New Frontiers in Summarization · doi:10.18653/v1/d19-5409 · Cited by 424 · see citing works
♫ MuSiQue: Multihop Questions via Single-hop Question Composition
Trivedi, H., Balasubramanian, N., Khot, T., & Sabharwal, A. (2022) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00475 · Cited by 283 · see citing works
LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Models
Jiang, H., Wu, Q., Lin, C.-Y., Yang, Y., & Qiu, L. (2023) · arXiv · Cited by 168 · see citing works
Generation-Augmented Retrieval for Open-Domain Question Answering
Mao, Y., He, P., Liu, X., Shen, Y., Gao, J., Han, J., & Chen, W. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.316 · Cited by 146 · see citing works
Fine-Tuned Transformers Show Clusters of Similar Representations Across Layers
Phang, J., Liu, H., & Bowman, S. R. (2021) · BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP (BlackboxNLP) · doi:10.18653/v1/2021.blackboxnlp-1.42 · Cited by 22 · see citing works
Optimizing LLM Queries in Relational Data Analytics Workloads
Liu, S., Biswal, A., Kamsetty, A., Cheng, A., Schroeder, L. G., Patel, L., Cao, S., Mo, X., Stoica, I., Gonzalez, J. E., et al. (2024) · arXiv · doi:10.48550/arxiv.2403.05821 · Cited by 8 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Agentic Context Engineering: Evolving Contexts for Self-Improving Language Models
Zhang, Q., Hu, C., Upasani, S., Ma, B., Hong, F., Kamanuru, V., Rainton, J., Wu, C., Ji, M., Li, H., et al. (2025) · arXiv · doi:10.48550/arxiv.2510.04618 · Cited by 3 · see citing works
“Meanwhile, advances in long-context LLMs (Peng et al., 2023) and context-efficient inference such as KV cache reuse (Gim et al., 2024; Yao et al., 2025) are making context-based approaches increasingly practical for deployment.”
Self-citationCompactor: Calibrated Query-Agnostic KV Cache Compression with Approximate Leverage Scores
Chari, V., & Van Durme, B. (2025) · arXiv · doi:10.48550/arxiv.2507.08143 · Cited by 1 · see citing works
“However, in the query-agnostic regime, their performance suffers greatly [25, 7].”
SmoothAgent: Efficient Long-Horizon LLM-Based Agent Serving with Lookahead Context Engineering
Pan, Z., Wang, Q., Hu, Z., Chen, C., Guan, Y., Zhou, Y., Swanson, S., & Ding, Y. (2026) · arXiv · doi:10.48550/arxiv.2607.00151
“Lotsofwork [16, 34, 42, 49, 54, 72] study efficient KV cache management to improve system performance.”
SparseX: Efficient Segment-Level KV Cache Sharing for Interleaved LLM Serving
Zhang, Q., Chen, K., Liao, N., Lin, Z., Tang, B., Xiong, F., Li, Z., & Wang, X. (2026) · arXiv · doi:10.48550/arxiv.2606.01751
“For example, CacheBlend [6] targets RAG scenarios, concatenates multiple cached knowledge chunks into a new request, and selectively recomputes KV Cache tensors for a subset of tokens to restore cross-chunk interactions.”
Self-citationPEEK: Predictive Queue-Informed KV Cache Management for LLM Serving
Xie, B., Wang, Z., Tanaka, M., & Zheng, Z. (2026) · arXiv · doi:10.48550/arxiv.2607.02525
“…[Juravsky et al., 2024] (fused shared-prefix kernel), CachedAttention [Gao et al., 2024] (session-scoped state), CacheBlend [Yao et al., 2024] (block reuse across retrieved chunks), CacheGen [Liu et al., 2024] (compressed KV streaming), LMCache [LMCache contributors, 2025] (distributed shared KV).”
Unified AI Gateway: A Framework for Joint Model Routing and KV Cache Management
Lu, J., Zhang, X., & Shao, Y. (2026) · arXiv · doi:10.48550/arxiv.2609.06940
“…matching across lexically different prompts [8], [14], (iii) non-prefix segment reuse with positional and cross-segment interaction handling [15], [16], (iv) multi-agent adaptation for cache off-sets and memory-position drift [17], [18], and (v) dynamic heat management for prioritization…”
InfluentialZhang, L. (2026) · arXiv · doi:10.48550/arxiv.2606.13361
“The substrate for this is mature: reuse [9, 1, 7], persistence [6, 4], transfer [3, 2], and compression [5, 8]; what remains open is the cross-party economic layer (pricing, invalidation on model upgrade, provenance, fallback on miss) and the standardization that exact, model-bound reuse implies.”
InfluentialContinuum: Efficient and Robust Multi-Turn LLM Agent Scheduling with KV Cache Time-to-Live
Li, H., He, R., Mang, Q., Zhang, Q., Mao, H., Chen, X., Zhou, H., Zhang, H., Cheung, A., Gonzalez, J., et al. (2025) · arXiv · doi:10.48550/arxiv.2511.02230
Self-citationAgentic Plan Caching: Test-Time Memory for Fast and Cost-Efficient LLM Agents
Zhang, Q., Wornow, M., Wan, G., & Olukotun, K. (2025) · arXiv · doi:10.48550/arxiv.2506.14852
“…caching has emerged as a popular approach, with two primary implementations: Context caching ( e.g., KV cache reuse and prompt caching [20, 60, 62]) stores internal model states to speed up subsequent generations, while semantic caching [10, 46, 1] stores and reuses (input, output) pairs to…”
InfluentialSelf-citationCoDec: Prefix-Shared Decoding Kernel for LLMs
Wang, Z., Ning, R., Fang, C., Zhang, Z. J., Lin, X., Ma, S., Zhou, M., Li, X., Wang, Z., Huan, C., et al. (2026) · Proceedings of the ACM on Management of Data · doi:10.1145/3802028
Asynchronous Speculative Sentence Decoding with Step-Level Alignment Classification
Hu, X., Li, Y., Ye, K., & Gao, X. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-36493-7_9
Allahwerdi, I., Soomro, G. M., Katta, R., Krayem, S., & Chramcov, B. (2026) · Machine Learning and Knowledge Extraction · doi:10.3390/make8110346
- Authors
- Yao, J., Li, H., Liu, Y., Ray, S., Cheng, Y., Zhang, Q., Du, K., Lu, S., & Jiang, J.
- Affiliations
- University of Chicago (US)
- Stanford University (US)
- Microsoft (United States) (US)
- ORCID
- Venue
- ACM Transactions on Computer Systems
- Published
- 2026-03-23
- ISSN
- 0734-2071 (ISSN-L)
- 1557-7333
- OpenAlex
- W7140127569
- Cited by
- 3
- References
- 9
- Citations by year
- 2026: 3
- FWCI
- 10.39
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- ACM Transactions on Computer Systems (publishedVersion, cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Hardware and Architecture › Parallel Computing and Optimization Techniques
- Keywords
- large language model serving, KV cache reuse, retrieval-augmented generation, knowledge fusion, time-to-first-token, inference throughput, cross-attention, LLM inference acceleration
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
