CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusion

A PIC method first applies RoPE rotation to align cached Keys with the target positions in the new request, then computes key differences between the rotated cached values and freshly computed values to identify important positions.

“The key idea is to correct for positional encoding differences: a PIC method first applies RoPE rotation to align cached Keys with the target positions in the new request, then computes key differences between the rotated cached values and freshly computed values to identify important positions”

PIC recovers sharing beyond the prefix at the cost of additional compute for rotation and recomputation, and a minor accuracy trade-off at positions that are reused without recomputation.

“This recovers sharing beyond the prefix at the cost of additional compute for rotation and recomputation, and a minor accuracy trade-off at positions that are reused without recomputation.”

Position-independent caching (PIC) methods remove the positional constraint and can reuse blocks at arbitrary offsets, but they still process each request separately: in an N-agent round the same shared blocks trigger N separate reuse passes, paying per-request overhead.

“Positionindependent caching (PIC) methods [10, 34-36] remove this constraint and can reuse blocks at arbitrary offsets, but they still process each request separately: in an 𝑁 -agent round the same shared blocks trigger 𝑁 separate reuse passes, paying per-request overhead for what is fundamentally a roundlevel sharing opportunity.”

Position-independent caching (PIC) methods remove the positional constraint by recovering shared blocks at arbitrary offsets.

“Position-independent caching (PIC) methods [10, 34-36] remove this constraint by recovering shared blocks at arbitrary offsets.”

The current prototype uses CacheBlend's selective recomputation as the default backend, but any PIC method that accepts a set of token positions and returns corrected K/V tensors can serve as a drop-in replacement through an adapter interface.

“The current prototype uses CacheBlend [35]'s selective recomputation as the default backend, but any PIC method that accepts a set of token positions and returns corrected K/V tensors can serve as a drop-in replacement through an adapter interface.”

CacheBlend adds selective recomputation to restore accuracy at important positions.

“CacheBlend [35] further adds selective recomputation to restore accuracy at important positions.”

CacheBlend selectively recomputed boundary tokens and reduced TTFT by 2.2-3.3× while improving throughput by 2.8-5× with negligible quality loss.

“CacheBlend selectively recomputed boundary tokens and reduced TTFT by 2.2-3.3 × while improving throughput by 2.8-5 × with negligible quality loss [15].”

Non-prefix segment reuse with positional and cross-segment interaction handling is required.

“non-prefix segment reuse with positional and cross-segment interaction handling [15], [16]”

Multi-turn chat and RAG use representative hit rates for persistent dialogue and non-prefix document reuse.

“Multi-turn chat and retrieval-augmented generation (RAG) use representative hit rates for persistent dialogue and non-prefix document reuse [15], [62].”

CacheBlend updates a reused KV cache, but it still feeds the KV cache to the same LLM that generated the KV cache, and it updates the KV cache of certain tokens rather than certain layers.

“For example, CacheBlend [97] updates a reused KV cache, but it still feeds the KV cache to the same LLMthat generated the KV cache. Moreover, it updates the KV cache of certain tokens, rather than certain layers like in DroidSpeak.”

Advances in long-context LLMs and context-efficient inference such as KV cache reuse are making context-based approaches increasingly practical for deployment.

“Meanwhile, advances in long-context LLMs (Peng et al., 2024) and context-efficient inference such as KV cache reuse (Gim et al., 2024; Yao et al., 2025) are making context-based approaches increasingly practical for deployment.”