CoChat
Sign Up
Cited by 22Topic ModelingParallel Computing and Optimization TechniquesNatural Language Processing Techniques

Cache-Craft: Managing Chunk-Caches for Efficient Retrieval-Augmented Generation

Open access
Agarwal, Shubham, Sundaresan, Sai, Mitra, Subrata, Mahapatra, Debabrata, Gupta, Archit, Sharma, Rounak, Kapu, Nirmal Joshua, Yu, Tong +1 more·2025·Proceedings of the ACM on Management of Data
DOI 10.1145/3725273
Open this paper in CoChat — free accountOpen PDF

Abstract

Retrieval-Augmented Generation (RAG) is often used with Large Language Models (LLMs) to infuse domain knowledge or user-specific information. In RAG, given a user query, a retriever extracts chunks of relevant text from a knowledge base. These chunks are sent to an LLM as part of the input prompt. Typically, any given chunk is repeatedly retrieved across user questions. However, currently, for every question, attention layers in LLMs fully compute the Keys and Values (KVs) repeatedly for the input chunks, as state-of-the-art methods cannot reuse KV-caches when chunks appear at arbitrary locations or with arbitrary contexts. Naive reuse leads to output quality degradation. This leads to potentially redundant computations on expensive GPUs and increases latency. In this work, we propose Cache-Craft , a system for managing and reusing precomputed KVs corresponding to the text chunks (which we call chunk-caches ) in RAG-based systems. We present how to identify chunk-caches that are reusable, how to efficiently perform a small fraction of recomputation to fix the cache and maintain output quality, and how to efficiently store and evict chunk-caches in the hardware for maximizing reuse while masking any overheads. With real production workloads as well as synthetic datasets, we show that Cache-Craft reduces redundant computation by 51% over SOTA prefix-caching and 75% over full recomputation. Additionally, with continuous batching on a real production workload, we get a 1.6× speed up in throughput for both the LLama-3-8B and 70B models and a 2.1× and 2× reduction in end-to-end response latency respectively, compared to prefix-caching, while maintaining generation quality.

Integrity signals

Registry

Crossref · 10.1145/3725273

Cites

75 of 75 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

11 of 22 cited sources checked · 3 supported · 3 flagged · accuracy not yet measured

  • Premisecitation

    Most prior KV reuse works assume prefix-match, which is impractical for RAG-systems.

    “Most of these assume prefix-match [29,41,53,54,85], which is impractical for RAG-systems.”
    Quote in the paperSubstantial claim2 of 5 sources checked
    • cites [29] 29. Bin Gao, Zhuomin He, Puru Sharma, Qingxuan Kang, Djordje Jevdjic, Junbo Deng, Xingkun Yang, Zhou Yu, and Pengfei Zuo. 2024. AttentionStore: Cost-effective Attention Reuse across Multi-turn Conversations in Large Language Model Serving. arXiv preprint arXiv:2403.19708 (2024). source

      Section 5 (Related Work; page unavailable) says RadixAttention and ChunkAttention share KV caches between tasks with the same prompt prefixes. It does not establish that most prior KV-reuse works require prefix matching or discuss whether prefix matching is impractical for RAG systems. (LLM, conf=0.85)

      Experimental — accuracy not yet measured
    • cites [41] 41. Chao Jin, Zili Zhang, Xuanlin Jiang, Fangyue Liu, Xin Liu, Xuanzhe Liu, and Xin Jin. 2024. RAGCache: Efficient Knowledge Caching for Retrieval-Augmented Generation. arXiv preprint arXiv:2404.12457 (2024). source
    • cites [53] 53. Shu Liu, Asim Biswal, Audrey Cheng, Xiangxi Mo, Shiyi Cao, Joseph E Gonzalez, Ion Stoica, and Matei Zaharia. 2024. Optimizing llm queries in relational workloads. arXiv preprint arXiv:2403.05821 (2024). source

      The paper does use shared prompt prefixes for KV-cache reuse (LLM inference), but it does not describe prefix matching as impractical for RAG. Instead, its RAG-query experiment reports that reordering retrieved contexts improves prefix reuse and speeds execution (User Prompt – Multi-LLM Invocation; page unavailable in our copy). It does not establish that most prior KV-reuse works assume prefix matching. (LLM, conf=0.83)

      Checked by a reasoning model — may be wrong.

      Experimental — accuracy not yet measured
    • cites [54] 54. Yuhan Liu, Hanchen Li, Kuntai Du, Jiayi Yao, Yihua Cheng, Yuyang Huang, Shan Lu, Michael Maire, Henry Hoff- mann, Ari Holtzman, et al. 2023. Cachegen: Fast context loading for language model applications. arXiv preprint arXiv:2310.07240 (2023). source
    • cites [85] 85. Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Jeff Huang, Chuyue Sun, Cody_Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E Gonzalez, et al. 2023. Efficiently Programming Large Language Models using SGLang. (2023).

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Premisecitation

    State-of-the-art methods cannot reuse KV-caches when chunks appear at arbitrary locations or with arbitrary contexts.

    “as state-of-the-art methods cannot reuse KV-caches when chunks appear at arbitrary locations or with arbitrary contexts.”
    Quote in the paperSubstantial claim1 of 3 sources checked
    • cites [43] 43. Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Symposium on Operating Systems Principles. 611-626. source

      The Introduction says existing serving systems cannot share KV caches between sequences because they store them in separate contiguous spaces. Section 3 also says generated KV caches depend on context and position. The paper does not specifically assess reuse of chunks appearing at arbitrary locations or in arbitrary contexts, so the claim is broader than its findings. (LLM, conf=0.83)

      Experimental — accuracy not yet measured
    • cites [54] 54. Yuhan Liu, Hanchen Li, Kuntai Du, Jiayi Yao, Yihua Cheng, Yuyang Huang, Shan Lu, Michael Maire, Henry Hoff- mann, Ari Holtzman, et al. 2023. Cachegen: Fast context loading for language model applications. arXiv preprint arXiv:2310.07240 (2023). source
    • cites [85] 85. Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Jeff Huang, Chuyue Sun, Cody_Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E Gonzalez, et al. 2023. Efficiently Programming Large Language Models using SGLang. (2023).

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Findingown data

    In Sys-X, prefill accounts for up to 55.4% of total inference time.

    “In Sys-X, it accounts for up to 55.4% of total inference time.”
    • Also stated in: § 2.2 Prefill Dominates Decode in RAG · plain
      “For Sys-X, prefill accounts for 55.4% of total time and 19.3x decode operations.”
    Quote in the paperSubstantial claim
  • Findingown data

    Output deviations increase with higher CCI and higher 1-β', indicating greater fixing requirements.

    “Fig. 12 shows how output deviations increase with higher 𝐶𝐶𝐼 and higher 1 -𝛽 ′ , indicating greater fixing requirements.”
    Quote in the paperSubstantial claim
  • Findingown data

    The number of prefill tokens is significantly more than decode: 30k prefill tokens for 600 decode tokens on average.

    “Fig. 1 shows a disparity in the number of tokens between prefill and decode: 30k prefill tokens for 600 decode tokens on average.”
    Quote in the paperSubstantial claim
  • Classified as probably not a substantial claim. 16
  • May combine two claims. 70
  • The quote only partly states this claim. 1

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Prompt Cache reuses KV-caches at different positions but relies on a rigid prompt structure, leading to poor quality when the structure changes.

    “Prompt Cache [31] reuses KV-caches at different positions but relies on a rigid prompt structure, leading to poor quality when the structure changes.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites [31] 31. In Gim, Guojun Chen, Seung-seob Lee, Nikhil Sarda, Anurag Khandelwal, and Lin Zhong. 2024. Prompt cache: Modular attention reuse for low-latency inference. Proceedings of Machine Learning and Systems 6 (2024), 325-338. source

      Section 3.1 (p. 4) confirms that Prompt Cache reuses attention states across prompts derived from a schema, but says users can select or replace modules at runtime and that output quality is not affected when relative token positions are preserved. Section 5.6 (p. 10) describes more complicated prompt structures and reports maintained output quality. The paper therefore contradicts the claim that changing the structure leads to poor quality. (LLM, conf=0.87)

      Checked by a reasoning model — may be wrong.

      Experimental — accuracy not yet measured

  • May combine two claims.

    Premisecitation

    Prompt Cache enables KV cache reuse at arbitrary positions but struggles to maintain satisfactory generation quality due to inaccuracies in positional encoding and a lack of consideration for cross-attention.

    “Although Prompt Cache [31] enables KV cache reuse at arbitrary positions, it struggles to maintain satisfactory generation quality due to inaccuracies in positional encoding and a lack of consideration for cross-attention.”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites [31] 31. In Gim, Guojun Chen, Seung-seob Lee, Nikhil Sarda, Anurag Khandelwal, and Lin Zhong. 2024. Prompt cache: Modular attention reuse for low-latency inference. Proceedings of Machine Learning and Systems 6 (2024), 325-338. source

      The paper directly contradicts the claim that Prompt Cache struggles to maintain generation quality: §5.3 (p. 8) reports output accuracy comparable to the baseline. It does identify position dependence (§3.1, p. 3) and says masking attention across modules can degrade quality when modules are semantically dependent (§3.3, p. 6), but it does not attribute generally unsatisfactory quality to inaccurate positional encoding or an unaddressed lack of cross-module attention. (LLM, conf=0.88)

      Checked by a reasoning model — may be wrong.

      Experimental — accuracy not yet measured

  • Classified as probably not a substantial claim.

    Premisecitation

    RAG systems typically operate on a finite knowledge base.

    “RAG systems typically operate on a finite knowledge base [46].”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites [46] 46. Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. 2020. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in Neural Information Processing Systems 33 (2020), 9459-9474. source

  • May combine two claims.

    Findingcitation

    Fig. 2 shows prefill time increases with input token length across different batch sizes for LLaMA-3-70B using vLLM with 4 NVIDIA A100-80GB GPUs, reaching up to 76 seconds for 32k-token sequences at a batch size 8.

    “Fig. 2 shows prefill time increases with input token length across different batch sizes for LLaMA-3-70B [72] using vLLM [43] with 4 NVIDIA A100-80GB GPUs, reaching up to 76 seconds for 32𝑘-token sequences at a batch size 8.”
    Quote in the paperSubstantial claim2 of 2 sources checked
    • cites [72] 72. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 (2023). source

      The citation is attached to the model name LLaMA-3-70B, not to the citing paper’s prefill-time measurement. The Abstract (p. 1) identifies models ranging from 7B to 65B parameters; searches for “LLaMA-3-70B” and “70B LLaMA 3 model” found no passage identifying the cited paper as a source for that model. (LLM, conf=0.76)

      Experimental — accuracy not yet measured
    • cites [43] 43. Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Symposium on Operating Systems Principles. 611-626. source

  • May combine two claims.

    Premisecitation

    A chunk containing more tokens is more reusable because tokens closer to each other exhibit stronger attention due to positional embeddings, compared to distant tokens from other chunks.

    “In fact, a chunk containing more tokens is more reusable because tokens closer to each other exhibit stronger attention due to positional embeddings, compared to distant tokens from other chunks [70].”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites [70] 70. Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. 2024. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing 568 (2024), 127063. source

      Section 3.3 (p. 5) says RoPE’s query–key inner product decays as relative token distance increases, supporting the proximity-based part of the claim. It does not address chunks or establish that chunks containing more tokens are more reusable; its finding is also about inner products, not a guarantee that every closer pair receives stronger attention. (LLM, conf=0.88)

      Experimental — accuracy not yet measured

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    According to prior studies, a ROUGE-L F1 score ≥ 0.6 is considered good, and a score ≥ 0.8 is considered almost indistinguishable from the original answer.

    “Note, according to several prior studies [12,51], a ROUGE-L F1 score ≥ 0.6 is considered good, and a score ≥ 0.8 is considered almost indistinguishable from the original answer.”
    Quote in the paperNot a substantial claim1 of 2 sources checked
    • cites [12] 12. Anthony Chen, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. 2019. Evaluating question answering evaluation. In Proceedings of the 2nd workshop on machine reading for question answering. 119-124. source

      Section 2 (Metrics) explains how ROUGE-L is computed, and Section 5.3 reports correlations with human judgments, but neither assigns quality labels to ROUGE-L F1 scores. Searches for the 0.6 and 0.8 thresholds and for the claimed descriptions found no supporting passage in our copy; the reported correlations are not individual answer scores. (LLM, conf=0.76)

      Experimental — accuracy not yet measured
    • cites [51] 51. Chin-Yew Lin and FJ Och. 2004. Looking for a few good metrics: ROUGE and its evaluation. In Ntcir workshop.

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    A threshold of 0.7 is considered good for semantic similarity.

    “while a threshold of 0.7 is considered good for semantic similarity [48].”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites [48] 48. Shuo Li, Sangdon Park, Insup Lee, and Osbert Bastani. 2024. TRAQ: Trustworthy Retrieval Augmented Question Answering via Conformal Prediction. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), Kevin Duh, Helena Gomez, and Steven Bethard (Eds.). Association for Computational Linguistics, Mexico City, Mexico, 3799-3821. https://doi.org/10.18653/v1/2024.naacl-long.210 source

  • Probably not a substantial claim: may not pass S4 (based).

    Findingnone given

    In RAG, a given chunk is typically repeatedly retrieved across user questions.

    “Typically, any given chunk is repeatedly retrieved across user questions.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    A small subset of tokens in a chunk significantly impacts the CCI score, and recomputing these critical tokens reduces output deviation.

    “We have observed that a small subset of tokens in a chunk significantly impacts the 𝐶𝐶𝐼 score (Fig. 14).Also, recomputing these critical tokens reduces output deviation (Fig. 15).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Prefix-Cache offers exact answers, but due to low prefix match rates, 80-95% of tokens go through regular KV-computation, leading to very low compute savings.

    “Prefix-Cache offers exact answers, but due to low prefix match rates, 80-95% of tokens go through regular KV-computation, leading to very low compute savings.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Prefill-H2O shows only a modest 2-10% improvement over Full-Cache but struggles with multi-hop tasks.

    “Prefill-H2O shows only a modest 2-10% improvement over Full-Cache but struggles with multi-hop tasks.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    The 5-tuple retrieval hit rate is significantly lower than individual chunk hit rates and does not follow the power law.

    “Fig. 6c shows that this hit rate is significantly lower compared to those of the individual chunks in Fig. 6a.However, a high hit rate is crucial for cache utilization.Moreover, unlike Fig. 6a, the distribution in Fig. 6c does not follow the power law”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    When inter-chunk attention weights and intra-chunk attention weights are highly overlapping, naive reuse of stale KV-cache results in a wrong output; when they are less overlapping, the precomputed KV-cache can lead to the right answer.

    “We observe in Fig. 10 that when the values of intra-chunk attention weights (from 𝑄 𝐶 2 𝐾 𝑇 𝐶 2 ) and inter-chunk attention weights (from 𝑄 𝐶 2 𝐾 𝑇 𝐶 1 ) are highly overlapping, naive reuse of stale KV-cache results in a wrong output.Whereas if they are less overlapping, the precomputed KV-cache can lead to the right answer in Fig. 9.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    For Sys-X, 75% of the retrieved chunks for a query were reprocessed, amounting to over 12B tokens in a month.

    “For Sys-X, 75% of the retrieved chunks for a query were reprocessed, amounting to over 12B tokens in a month.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Using Full-Cache incurs no recomputation but yields low quality, with ROUGE score dropping to 0.65 for multi-hop QA datasets like 2wikiMQA and MuSiQue.

    “Using Full-Cache incurs no recomputation but yields low quality, with the ROUGE score dropping to 0.65 for multi-hop QA datasets like 2wikiMQA and MuSiQue.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Cache-Craft outperforms Full-Cache by up to 50% in short-QA and True/False tasks, achieving ROUGE score of 0.87 compared to 0.59.

    “Fig. 21 further shows that Cache-Craft outperforms Full-Cache by up to 50% in short-QA and True/False tasks, achieving ROUGE score of 0.87, compared to 0.59.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    In production workloads, chunk-cache size grows with question diversity, but a small set of chunks remains crucial.

    “In production workloads, the chunk-cache size grows with question diversity, but a small set of chunks remains crucial.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    The top 5% of chunks are accessed by 60% of the requests in both Sys-X and MuSiQue, and 40% of requests in 2wikiMQA.

    “The top 5% of chunks are accessed by 60% of the requests in both Sys-X and MuSiQue, and 40% requests in 2wikiMQA.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Cache-Craft achieves a ROUGE score of 0.87 with only 20% token recomputation on Sys-X, outperforming Full-Cache reuse (0.59) and other recomputation strategies by about 20-30%.

    “As shown in Fig. 20, Cache-Craft achieves a ROUGE score of 0.87 with only 20% token recomputation, outperforming Full-Cache reuse (0.59) and other recomputation strategies by about 20-30%.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Cache-Craft reduces redundant computation by 51% over SOTA prefix-caching and 75% over full recomputation.

    “With real production workloads as well as synthetic datasets, we show that Cache-Craft reduces redundant computation by 51% over SOTA prefix-caching and 75% over full recomputation.”
    • Also stated in: § Cache-Craft: · plain
      “We show that it achieves a 51% reduction in GPU computation costs for production workloads compared to prefix-caching ( §5.4).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    In Sys-X, most chunk reuse occurs across users (94%), with reuse within a session at 55% and across sessions at 67%.

    “In Sys-X, most chunk reuse occurs across users (94%), with reuse within a session at 55% and across sessions at 67%.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingnone given

    For 5 RAG chunk-caches corresponding to a request in Sys-X, loading time is 0.03s from CPU and 0.59s from SSD.

    “For example, the loading time required for 5 RAG chunk-caches corresponding to a request in Sys-X takes 0.03s from CPU and 0.59s from SSD.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    In production workloads, exact prefix caching applies to only 8% of requests and 18% of total prefill tokens.

    “In our production workloads, we found that exact prefix caching applies to only a small fraction (8%) of requests and 18% of total prefill tokens.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Under continuous batching on a real production workload, Cache-Craft reduces end-to-end response latency by 2.1× for LLaMA-3-8B and 2× for LLaMA-3-70B compared to prefix-caching.

    “and a 2.1× and 2× reduction in end-to-end response latency respectively, compared to prefix-caching, while maintaining generation quality.”
    • Also stated in: § Cache-Craft: · plain
      “Under continuous batching through ORCA for Sys-X, Cache-Craft improves throughput by 1.6× and reduces end-to-end response latency by 2.1× for LLaMA-3-8B model and for LLaMA-3-70B, it provides a 1.6× speedup in throughput and 2× reduction in end-to-end response latency compared to prefix-caching ( §5.3).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    For Sys-Y, prefill accounts for 76% of total time and 46× decode operations.

    “Sys-Y takes 76% of the time and 46× the operations, while LMSys uses 22% time and 4.4× operations.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    In Sys-X, a typical queue wait time is 0.32s, allowing preloading chunks from either CPU or SSD without impacting latency significantly.

    “In Sys-X, a typical queue wait time is 0.32s, allowing for preloading chunks from either CPU or SSD without impacting latency significantly.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    In the user study, Full-Cache achieved significantly lower scores (30% on 2wikiMQA, 53% on SQuAD), whereas Cache-Craft consistently outperformed it (71% on 2wikiMQA, 79% on SQuAD).

    “As illustrated in Table 2, Full-Cache achieved significantly lower scores (30% on 2wikiMQA, 53% on SQuAD), whereas Cache-Craft consistently outperformed it (71% on 2wikiMQA, 79% on SQuAD).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Set-Cache gives slightly more savings of 15-35% by making prefixes permutation invariant but with lower ROUGE scores due to incorrect contextualization between chunk-caches.

    “Set-Cache gives slightly more savings of 15-35% by making prefixes permutation invariant but with lower ROUGE scores due to incorrect contextualization between the chunk-caches.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    The focused-chunk early-termination approach reduces token recomputation by about 55% while maintaining similar output quality.

    “In §6, we observe that this approach reduces token recomputation by about 55% while maintaining similar output quality.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    For LMSys, prefill accounts for 22% of total time and 4.4× decode operations.

    “Sys-Y takes 76% of the time and 46× the operations, while LMSys uses 22% time and 4.4× operations.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    In the user study, for answers with ROUGE-L F1 scores ≥ 0.6 and ≥ 0.8, 81% and 93% of users gave a YES, respectively.

    “From our user study, we also analyzed this correlation and found that for answers with ROUGE-L F1 scores ≥ 0.6 and ≥ 0.8, 81% and 93% of users have given a YES, respectively.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Inter-attention scores vary during initial layers but after a certain number of layers settle into values that can segregate focused chunks from others.

    “We find that while the inter-attention scores vary during the initial layers, after a certain number of layers, they settle into values that can segregate the focused chunks from the others.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Prefix-Cache saves just 18% prefill tokens on Sys-X, proving ineffective.

    “We also see that Prefix-Cache also saves just 18% prefill tokens, proving ineffective.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Findingnone given

    Padding variable chunk sizes to align with 16-token blocks causes negligible output deviation.

    “Such padding causes negligible output deviation.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    By 3 months in Sys-X, chunk accesses increased to 24%, but 85% of requests still accessed just 14.8% of chunks, indicating minimal growth in the required chunk set.

    “By 3 months, chunk accesses increased to 24%, but 85% of requests still accessed just 14.8%, indicating minimal growth in the required chunk set.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    In both cases, 30% of tokens are recomputed, which maintains about 90% of the base ROUGE F1 score on average.

    “In both cases, 30% tokens are recomputed, which maintains about 90% of the base ROUGE F1 score on average.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    With 30% recomputation, Cache-Craft gives ROUGE-F1 scores around 0.9, which is ≈100% higher than Lingua2 (0.4) and MapReduce (0.5) for 70% compression.

    “In Table 1, it can be observed that with 30% recomputation, Cache-Craft gives ROUGE-F1 scores around 0.9, which is ≈100% higher than the scores for Lingua2 (0.4) and MapReduce (0.5), for 70% compression (i.e., comparable to 30% recomputation).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    In Sys-X, 20.6% of chunks were accessed over 1 month, with 85% of requests hitting only 13.5% of chunks.

    “For example, in Sys-X, 20.6% of chunks were accessed over 1 month, with 85% of requests hitting only 13.5% of chunks.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Recomputing 20% of tokens with LLaMA-3-8B improves ROUGE score by 30%, and 30% recomputation improves it by 42%.

    “In contrast, recomputing 20% of tokens with LLaMA-3-8B improves the ROUGE score by 30%, and further by 42% with 30% recomputation.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Cache-Craft offers the best trade-off between recomputation budget and quality, outperforming all baselines with a ROUGE F1 score of 0.87 using 20% recomputation for Sys-X.

    “Fig. 20 shows Cache-Craft offers the best trade-off between recomputation budget and quality, outperforming all baselines with a ROUGE F1 score of 0.87 using 20% recomputation for Sys-X”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    With 30% token recomputation, maintaining 90% of the base ROUGE F1 score on average, Cache-Craft still achieves a 1.6× speedup in throughput and a 2.1× reduction in response latency for LLaMA-3-8B, and for LLaMA-3-70B a 1.6× speedup in throughput and a 2× reduction in response latency under high load.

    “With 30% token recomputation, maintaining 90% of the base ROUGE F1 score on average, we still observe a 1.6× speedup in throughput and a 2.1× reduction in response latency for LLaMA-3-8B and for LLaMA-3-70B, the improvement is a 1.6× speedup in throughput and a 2× reduction in response latency under high load.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Fixing causality without adjusting RPE yields minimal improvement (0.22), while optimizing RPE alone achieves 0.665, which serves as the Full-Cache baseline.

    “Fixing causality without adjusting RPE yields minimal improvement (0.22) while optimizing RPE alone achieves 0.665, which serves as the Full-Cache baseline.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Disabling focus chunking raises recomputation to 70% with minimal quality gains.

    “Disabling focus chunking similarly raises recomputation to 70% with minimal quality gains”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    For a large number of requests in Sys-X, all necessary chunks for top-k=5 retrieval were already in the cache, leading to a 5 out of 5 hit-rate.

    “It can be observed that for a large number of requests, all the necessary chunks were already in the cache -leading to a 5 out of 5 hit-rate.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Reusing caches from nonprefix chunks significantly degrades performance, resulting in a ROUGE F1 score of 0.15 when neither RPE nor causality is fixed.

    “In Table 3, we observe that reusing caches from nonprefix chunks significantly degrades performance, resulting in a ROUGE F1 score of 0.15 when neither RPE nor causality is fixed.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    User study results show 78.9% user acceptance for Cache-Craft versus 83.7% for vanilla vLLM with exact computation using LLaMA-3-8B.

    “This is also supported by our user study results, which show 78.9% user acceptance for Cache-Craft versus 83.7% for vanilla vLLM with exact computation using LLaMA-3-8B (Table 2).”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Cache-Craft reduces latency by 1.6× and 2.3× compared to Full-Recomp for LLaMA-3-8B and LLaMA-3-70B, respectively, with batch-size=4 and sequence length of 8192 tokens.

    “Cache-Craft reduces latency by 1.6× and 2.3× compared to Full-Recomp for LLaMA-3-8B and LLaMA-3-70B, respectively, and with batch-size= 4 and sequence length of 8192 tokens.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Recomputing attention scores for only 30% of carefully selected tokens while using precomputed caches for the rest achieves 93% of the user score of full attention computation, while significantly improving performance.

    “Our evaluations in Fig. 20 and the qualitative user study in Table 2 show that recomputing attention scores for only 30% of carefully selected tokens, while using precomputed caches for the rest, achieves 93% of the user score of full attention computation, while significantly improving performance.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingreasoning

    With an average queue wait time of 0.32s for Sys-X, asynchronous preloading eliminates CPU overhead and reduces SSD overhead to 0.27s.

    “With an average queue wait time of 0.32s (for Sys-X), asynchronous preloading eliminates CPU overhead and reduces SSD overhead to 0.27s, as 0.32s overlaps with queue time.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingreasoning

    CPU loading overhead is 0s for Async and Layer because loading time is already less than queue time.

    “CPU loading overhead is 0s for Async and Layer because loading time is already less than queue time.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Cache-Craft outperforms all baselines across different sequence lengths, and for 16k token length it is 1.7× faster than Full-Recomp.

    “We can see Cache-Craft outperforms all baselines across different sequence lengths, and for 16k token length it is 1.7× faster than Full-Recomp.”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: For Prefix-Cache, 60% of prefill tokens have a prefix match.

  • May combine two claims.

    Findingown data

    On average, only 18% exact prefix match occurs for requests in Sys-X, rendering Prefix-Cache ineffective.

    “This is because, on average, only 18% exact prefix match occurs for the requests received in the system, rendering Prefix-Cache ineffective.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingreasoning

    Cache-Craft achieves about 2.5× speedup for LLaMA-3-8B in TTFT latency over Prefix-Cache by recomputing 39% of tokens while maintaining 90% of the original quality.

    “We observe about 2.5× speedup for LLaMA-3-8B in TTFT latency over Prefix-Cache by recomputing 39% of tokens while maintaining 90% of the original quality.”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: For Prefix-Cache, 60% of prefill tokens have a prefix match.

  • May combine two claims.

    Findingreasoning

    Cache-Craft stores chunks independently, allowing it to store more chunks in HBM and achieve higher cache hit rates by reusing chunks in different combinations.

    “Cache-Craft stores chunks independently, allowing to store more chunks in HBM and achieving higher cache hit rates by reusing chunks in different combinations.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Prefix caching has low hit rates and high memory overhead.

    “Unlike prefix caching, which has low hit rates and high memory overhead (Fig. 5a)”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Prefill-H2O showed moderate improvement (52% on 2wikiMQA, 67% on SQuAD), outperforming Full-Cache but still lagging behind Cache-Craft.

    “We also tested an alternative recomputation strategy, Prefill-H2O, which showed moderate improvement (52% on 2wikiMQA, 67% on SQuAD), outperforming Full-Cache but still lagging behind Cache-Craft.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Cache-Craft achieves up to a 1.9× speedup in throughput and a 3.3× reduction in response latency under a heavy load of 240 QPM for LLaMA-3-8B, and for LLaMA-3-70B it provides a 2× speedup in throughput and a 3.3× reduction in response latency under a heavy load of 120 QPM with no recomputation over Prefix-Cache.

    “As shown in Fig. 22, Cache-Craft achieves up to a 1.9× speedup in throughput and a 3.3× reduction in response latency under a heavy load of 240 QPM (Queries per minute) for the LLaMA-3-8B model and for LLaMA-3-70B, it provides a 2× speedup in throughput and a 3.3× reduction in response latency under a similar heavy load of 120 QPM with no recomputation over Prefix-Cache.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    For large chunks (>883 tokens) in Sys-X, the intra-attention is 2.18x inter-attention on average, with only 23% of chunks being highly contextualized.

    “For large chunks (>883 tokens) in Sys-X, the intra-attention is 2.18x inter-attention on average, with only 23% of chunks being highly contextualized.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given

    In Sys-X, the range of sequences varies from 600 to 20000 tokens, averaging around 5000 tokens.

    “The range of sequences received by the system varies from 600 to 20000 tokens, averaging around 5000 tokens.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingreasoning

    For caches stored in GPU memory, there is no additional overhead for loading.

    “Note that for caches stored in GPU memory, there is no additional overhead for loading.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Removing β increases recomputation to 54% without improving quality.

    “Removing 𝛽 increases recomputation to 54% without improving quality”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning

    Processing the reprocessed tokens for Sys-X would require 9600 hours of GPU compute on LLaMA-3-70B using 8 A100 GPUs, costing approximately $50,000.

    “Processing these tokens would require 9600 hours of GPU compute on LLaMA-3-70B using 8 A100 GPUs, costing approximately $50,000.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationown data

    Cache-Craft offers the best trade-off as its points on Fig. 20 are the furthest towards the top-left (ideal) corner.

    “To summarize, Cache-Craft offers the best trade-off as its points on Fig. 20 are the furthest towards the top-left (ideal) corner.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Premisecitation

    In RAG pipelines, it is standard practice to retrieve a sufficient number of chunks from a large knowledge base and utilize the LLM to filter out irrelevant content for coherent answers.

    “In RAG pipelines, it is a standard practice to retrieve a sufficient number of chunks from a large knowledge base and utilize the LLM to filter out irrelevant content for coherent answers [77].”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [77] 77. Zhiruo Wang, Jun Araki, Zhengbao Jiang, Md Rizwan Parvez, and Graham Neubig. 2023. Learning to filter context for retrieval-augmented generation. arXiv preprint arXiv:2311.08377 (2023). source

  • May combine two claims.

    Premisereasoning

    Using more chunks can improve response quality but a longer context would lead to even longer TTFT.

    “Using more chunks can improve response quality but a longer context would lead to even longer TTFT.”
    Quote in the paperSubstantial claim

  • May combine two claims. The quote only partly states this claim.

    Interpretationown data

    Cache-Craft maintains generation quality while achieving its throughput and latency improvements.

    “while maintaining generation quality.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Prefix caching preserves output quality by maintaining causal attention.

    “While this approach preserves the output quality by maintaining causal attention [74], its practical utility is very limited in RAG systems.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [74] 74. A Vaswani. 2017. Attention is all you need. Advances in Neural Information Processing Systems (2017).

      Not checked in the free run.

  • Classified as probably not a substantial claim.

    Study design and conductnone given

    The study uses two real production RAG systems, Sys-X and Sys-Y, for workload characterization, motivation, and evaluations.

    “In this regard, we use two real production RAG systems, referred to as Sys-X and Sys-Y, for workload characterization, motivation, and evaluations.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Interpretationreasoning

    A high Cache Context Impact (CCI) for a cache indicates that the chunk is highly contextualized, reducing its potential for reuse unless the prefix context matches closely; a low CCI suggests the chunk is largely independent of its prefix context, making it more reusable across different contexts.

    “A high value of the 𝐶𝐶𝐼 for a cache indicates that the chunk is highly contextualized, reducing its potential for reuse unless the prefix context matches closely.Conversely, a low 𝐶𝐶𝐼 suggests that the chunk is largely independent of its prefix context, making it more reusable across different contexts.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Premisecitation

    Indiscriminate reuse of KV-caches from previously processed parts of the knowledge base can disrupt the relative positions of tokens, violating causal attention and degrading output quality.

    “Indiscriminate reuse of KV-caches from previously processed parts of the knowledge base can disrupt the relative positions of tokens, violating causal attention and degrading output quality [74].”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [74] 74. A Vaswani. 2017. Attention is all you need. Advances in Neural Information Processing Systems (2017).

      Not checked in the free run.

  • May combine two claims.

    Premisereasoning

    RAG-system-retrieved text chunks and their relative ordering are sensitive to the input question; a slight variation in the user question can result in different sets of chunks and ordering, rendering the prefix caching technique ineffective.

    “This is because the RAG-system-retrieved text chunks and their relative ordering are sensitive to the input question.A slight variation in the user question can result in different sets of chunks and ordering, rendering the prefix caching technique ineffective.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductown data

    The analysis of token composition in Sys-X uses data from 25 sessions of a proprietary system.

    “We analyze this in Fig. 5 using data from 25 sessions of a proprietary system, Sys-X.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Premisereasoning

    The latency of loading precomputed KV-caches into GPU memory must not negate the savings from bypassing recomputation.

    “The latency of loading precomputed KV-caches into GPU memory must not negate the savings from bypassing recomputation, requiring efficient system design and implementation.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Interpretationnone given

    A higher ā/b ratio indicates greater outside contextual influence on the chunk's KV.

    “A higher ā b ratio indicates greater outside contextual influence on the chunk's KV.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationown data

    Cache-Craft identifies and recomputes critical tokens distorted by prior contexts, enhancing performance and minimizing missing or incorrect facts.

    “Cache-Craft identifies and recomputes critical tokens distorted by prior contexts, enhancing performance and minimizing missing or incorrect facts.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning

    When prefill lengths are high, Cache-Craft can reduce TTFT significantly by reusing chunk-caches, avoiding quadratic computational complexity.

    “However, note that when prefill lengths are high (leading to spikes), Cache-Craft can reduce TTFT significantly as by reusing chunk-caches, it avoids quadratic computational complexity ( §2).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning

    While throughput is bounded by GPU capacity, latency improves significantly due to reduced queuing and computation load per request.

    “While throughput is bounded by GPU capacity, latency improves significantly due to reduced queuing and computation load per request.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationown data

    Cache-Craft selectively recomputes KV caches for a few contextualized tokens, producing outputs close to ideal.

    “To address such chunks, Cache-Craft selectively recomputes KV caches for a few contextualized tokens, producing outputs close to ideal ( §3.2).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning

    The improvement in TTFT reduction with model size is due to the reduction of the number of tokens computed by Cache-Craft in each attention layer, with gains increasing as the number of layers grows for larger models.

    “This improvement is due to the reduction of the number of tokens computed by Cache-Craft in each attention layer, with the gains increasing as the number of layers grows for larger models.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationown data

    Both β and focused chunk selection are important in optimizing recompute efficiency.

    “underscoring the importance of both 𝛽 and focused chunk selection in optimizing recompute efficiency.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Interpretationown data

    The spikes in TTFT are due to text chunks in Sys-X being subsections of user manuals and unequal in size.

    “Note, the spikes in TTFT are due to the fact that text chunks in Sys-X are subsections of the user manuals and are unequal in size.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationown data

    Cache-Craft enables efficient KV-cache reuse for RAG without compromising quality.

    “Cache-Craft enables efficient KV-cache reuse for RAG without compromising quality.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: TTFT metric is critical in production.
82 of 82

  • The authors state this as an assumption.

    Higher prefix overlap implies greater reusability.

    “Assuming that higher prefix overlap implies greater reusability, we calculate a Prefix Overlap Score 𝛽 for a chunk-cache of 𝐶 𝑖 corresponding to the current prompt sequence 𝑆 𝑛𝑒𝑤 as:”

  • The authors state this as an assumption.

    For Prefix-Cache, 60% of prefill tokens have a prefix match.

    “For Prefix-Cache, we scope generously by assuming 60% of prefill tokens have a prefix match, much higher than the 18% prefix match rate that we observed in production workloads for Sys-X.”

  • The paper relies on this without support.

    Traditional cache policies like LRU, LFU, or FIFO do not offer the capability to manage storage dynamically by prioritizing caches that maximize reusability while minimizing recomputation.

    “Traditional policies like LRU, LFU, or FIFO do not offer this capability.”

  • The paper relies on this without support.

    TTFT metric is critical in production.

    “(2) TTFT metric is critical in production.”

  • The paper relies on this without support.

    The instructions in the prompt are the same across all prompts.

    “The instructions in the prompt are the same across all prompts.”
5 of 5

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • Inter-chunk attention is defined in the forbidden causal direction

    Minor

    Chunk Ci precedes Cj when i < j, but the stated metric measures attention from Ci to Cj. Under the paper's causal-attention rule, those earlier tokens cannot attend to the later chunk, so this metric cannot measure the cross-chunk influence used by the cache-reuse method.

  • Intra-chunk attention sum selects future tokens

    Minor

    The sum restricts k < l, while akl is defined as attention from token k to token l. That selects attention to later tokens, contradicting both the description of this metric as attention to previous tokens and the stated causal rule.

  • The written attention mask does not enforce causality

    Minor

    Multiplying future-token logits by zero before softmax gives those positions a nonzero softmax weight, rather than excluding them. Thus the displayed computation does not provide the causal masking claimed immediately afterward.

3 of 3

Cites

Flagged first, then by citations

Research works printed but not linked: 15

  1. 10.5555/3691825.3691890 · DOI
  2. AI Anthropic. 2024. The claude 3 model family: Opus, sonnet, haiku. Claude-3 Model Card 1 (2024).
  3. Mingcong Han, Hanze Zhang, Rong Chen, and Haibo Chen. 2022. Microsecond-scale preemption for concurrent {GPU-accelerated} {DNN} inferences. In 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22). 539--558.
  4. Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out. 74--81.
  5. Chin-Yew Lin and FJ Och. 2004. Looking for a few good metrics: ROUGE and its evaluation. In Ntcir workshop.
  6. Chen-Yi Lu, Shubham Agarwal, Md Mehrab Tanjim, Kanak Mahadik, Anup Rao, Subrata Mitra, Shiv Kumar Saini, Saurabh Bagchi, and Somali Chaterji. 2024. RECON: Training-Free Acceleration for Text-to-Image Synthesis with Retrieval of Concept Prompt Trajectories. In European Conference on Computer Vision. Springer, 288--306.
  7. Jesse Mu, Xiang Li, and Noah Goodman. 2024. Learning to compress prompts with gist tokens. Advances in Neural Information Processing Systems 36 (2024).
  8. Yining Shi, Zhi Yang, Jilong Xue, Lingxiao Ma, Yuqing Xia, Ziming Miao, Yuxiao Guo, Fan Yang, and Lidong Zhou. 2023. Welder: Scheduling deep learning memory access via tile-graph. In 17th USENIX Symposium on Operating Systems Design and Implementation (OSDI 23). 701--718.
  9. Philippe Tillet. 2021. Triton: Open-source GPU programming for neural networks. https://openai.com/index/triton/.
  10. A Vaswani. 2017. Attention is all you need. Advances in Neural Information Processing Systems (2017).
  11. Ran Xu, Jinkyu Koo, Rakesh Kumar, Peter Bai, Subrata Mitra, Sasa Misailovic, and Saurabh Bagchi. 2018. {VideoChef}: Efficient Approximation for Streaming Video Processing Pipelines. In 2018 USENIX Annual Technical Conference (USENIX ATC 18). 43--56.
  12. Gyeong-In Yu, Joo Seong Jeong, Geon-Woo Kim, Soojeong Kim, and Byung-Gon Chun. 2022. Orca: A distributed serving system for {Transformer-Based} generative models. In 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22). 521--538.
  13. Haoyu Zhang, Ganesh Ananthanarayanan, Peter Bodik, Matthai Philipose, Paramvir Bahl, and Michael J Freedman. 2017. Live video analytics at scale with approximation and {Delay-Tolerance}. In 14th USENIX Symposium on Networked Systems Design and Implementation (NSDI 17). 377--392.
  14. Hong Zhang, Yupeng Tang, Anurag Khandelwal, and Ion Stoica. 2023. {SHEPHERD}: Serving {DNNs} in the wild. In 20th USENIX Symposium on Networked Systems Design and Implementation (NSDI 23). 787--808.
  15. Zhenyu Zhang Ying Sheng Tianyi Zhou Tianlong Chen Lianmin Zheng Ruisi Cai Zhao Song Yuandong Tian Christopher Ré Clark Barrett et al. 2024. H2o: Heavy-hitter oracle for efficient generative inference of large language models. Advances in Neural Information Processing Systems 36 (2024).

Cited by

Sorted by citations

Related

Authors
Agarwal, S., Sundaresan, S., Mitra, S., Mahapatra, D., Gupta, A., Sharma, R., Kapu, N. J., Yu, T., & Saini, S.
Affiliations
  • Adobe Systems (United States) (US)
  • Indian Institute of Technology Bombay (IN)
  • Indian Institute of Technology Kanpur (IN)
  • IIT Kanpur, Kanpur, India
ORCID
Venue
Proceedings of the ACM on Management of Data
Publisher
Association for Computing Machinery (ACM)
Published
2025-06-17
Volume
3
Issue
3
Pages
1-28
ISSN
2836-6573 (ISSN-L)
DOI
10.1145/3725273
OpenAlex
W4411403197
Semantic Scholar
8f323e932374e158cb4ceb4376c9b0ab2ac35a1f
Cited by
21
References
40
Citations by year
2026: 17 · 2025: 4
FWCI
8.45
Type
journal article
Language
en
Open access
Locations
  • Proceedings of the ACM on Management of Data (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Hardware and Architecture › Parallel Computing and Optimization Techniques
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
Keywords
retrieval-augmented generation, large language models, KV cache management, cache eviction, continuous batching, redundant computation reduction

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-11.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-11.

Cited sources are checked by gpt-6-sol.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review