CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Serving

Limited GPU memory capacity poses a significant challenge for caching large-scale prefixes.

“Insufficient GPU Memory: While GPUs excel in computation, their limited memory capacity poses a significant challenge for caching large-scale prefixes [13,48].”

SwiftCache, KVServe, CacheGen, and TraCT showed that reload, bandwidth-aware compression, streaming, and shared memory must be considered together.

“SwiftCache, KVServe, CacheGen, and TraCT showed that reload, bandwidth-aware compression, streaming, and shared memory must be considered together [27], [32], [76], [79].”

Complementary techniques compress or stream the cache, persist it across multi-turn conversations, share it across enterprise deployments, or evict it adaptively under memory pressure; these are orthogonal to where the cache should physically live in a multi-cloud fleet.

“Complementary techniques compress or stream the cache [39], persist it across multi-turn conversations [40], share it across enterprise deployments [59], or evict it adaptively under memory pressure [41, 42]; these are orthogonal to where the cache should physically live in a multi-cloud fleet.”

HCache, InfiniGen, CacheGen, MoonCake, and CachedAttention reduce the cost of long-context serving through recomputation, offloading, compression, or disaggregation.

“HCache [5], InfiniGen [13], CacheGen [19], MoonCake [30], and CachedAttention [4] reduce the cost of long-context serving through recomputation, offloading, compression, or disaggregation.”

The compression and transfer manager makes reusable KV caches deliverable under heterogeneous edge bandwidth and storage constraints.

“The compression and transfer manager makes reusable KV caches deliverable under heterogeneous edge bandwidth and storage constraints [27].”

Context caching relies on KV caches that are model-dependent and not easily transferable across different models.

“These KV caches are inherently model-dependent and not easily transferable across different models [56, 33]”

DroidSpeak differs from prior work in why and how to re-compute KV cache: it is designed for KV cache reuse across different LLMs, while prior work assumes a single LLM; none of the prior work chooses to re-compute or re-use a group of layers like DroidSpeak.

“DroidSpeak differs with prior work [31,32,61,97] in why and how to re-compute KVcache: DroidSpeak is designed for KV cache reuse across different LLMs, while prior work assumes a single LLM; due to the different purposes, none of the prior work chooses to re-compute or re-use a group of layers like in DroidSpeak.”

CacheGen [39] compresses and streams the cache; InfiniGen [41] and H2O [42] evict adaptively for long contexts.

“CacheGen [39] compresses and streams the cache; InfiniGen [41] and H2O [42] evict adaptively for long contexts.”