The DOI matches its DataCite record; 33 of the 33 references it cites were found in a registry or an index; 13 of the 19 sources its claims cite were checked for support.
ChunkAttention: Efficient Self-Attention with Prefix-Aware KV Cache and Two-Phase Partition
Open accessAbstract
Self-attention is an essential component of large language models (LLM) but a significant source of inference latency for long sequences. In multi-tenant LLM serving scenarios, the compute and memory operation cost of self-attention can be optimized by using the probability that multiple LLM requests have shared system prompts in prefixes. In this paper, we introduce ChunkAttention, a prefix-aware self-attention module that can detect matching prompt prefixes across multiple requests and share their key/value tensors in memory at runtime to improve the memory utilization of KV cache. This is achieved by breaking monolithic key/value tensors into smaller chunks and structuring them into the auxiliary prefix tree. Consequently, on top of the prefix-tree based KV cache, we design an efficient self-attention kernel, where a two-phase partition algorithm is implemented to improve the data locality during self-attention computation in the presence of shared system prompts. Experiments show that ChunkAttention can speed up the self-attention kernel by 3.2-4.8$\times$ compared to the state-of-the-art implementation, with the length of the system prompt ranging from 1024 to 4096.
Overview
AI snapshot
Written from the paper’s full text.
## TL;DR
ChunkAttention is a self-attention module for LLM inference that exploits shared system prompts by storing KV cache in a prefix tree and using a two-phase partition kernel. On A100 (80G), it reports 3.2–4.8× speedups over the state of the art when shared system prompts are 1024–4096 tokens long.
## Key findings
- In microkernel experiments on NVIDIA A100 (80G) with FP16, ChunkAttn outperformed PagedAttn* by 2.8–3.2× when shared prefix length \(n_s\) ranged from 1024 to 4096; for example, at \(n_p=1024, n_s=1024\), latency dropped from 154.46 µs to 56.00 µs.
- In end-to-end tests on OpenLlama2 7B with ChunkLlama, token rate improved by 3.2×, 2.7×, and 2.2× at \(n_c = 256, 512,\) and 1024 for one setting, and by 4.8×, 4.1×, 3.2×, and 2.4× for shared prefix lengths 512, 1024, 2048, and 4096 respectively.
- Table 4 reports that with \(n_p=1024, n_s=1024, n_c=512\), peak KV cache memory fell from 14.79 GB to 3.28 GB; with \(n_p=2048, n_s=2048, n_c=512\), it fell from 21.09 GB to 3.40 GB.
- The paper states that without shared system prompts, ChunkLlama shows no performance regression versus vLLM, and can achieve comparable throughput to the SOTA PagedAttention kernel.
## Methods
- The paper proposes Prefix-Aware KV Cache (PAKV), organizing KV cache as a prefix tree of chunked context tokens so matching prompt prefixes can share key/value tensors in memory at runtime.
- It adds Two-Phase Partition (TPP) for decoding: a chunk-first phase over shared chunks and a sequence-first phase over per-sequence chunks, using online softmax and specialized CUDA kernels.
- Evaluation includes both self-attention microkernel tests and end-to-end GPT-style serving tests on NVIDIA A100 (80G) with CUDA 11.8; microkernel settings include FP16, head dimension 128, 32 heads, and chunk size 64.
## Limitations
- The paper states that the shared system prompt must appear at the beginning of the sequence for KV sharing; if developers place it elsewhere, PAKV cannot save memory even when many tokens are common.
- The paper states that ChunkAttention’s best performance depends on low-level CUDA implementations and is tuned for common configurations such as 128 head dimension and hardware like NVIDIA A100, GeForce RTX 4090, and Intel Xeon CPU; other configurations require case-by-case tuning.
- Evident from the text: experiments are reported only on A100 (80G) and OpenLlama2 7B / Llama 7B-style settings, so broader generalization to other models and hardware is not directly shown in the paper.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
13 of 19 cited sources checked · 6 supported · 0 flagged · accuracy not yet measured
- Premisereasoning§ 1 Introduction
The Kwon et al. proposal has limitations: predefined system prompts are static and inflexible in frequent refreshes for large-scale deployments.
“The proposal has limitations: i) predefined system prompts are static and inflexible in frequent refreshes for largescale deployments since both application developers and the service provider are involved in the operation loop”
Quote in the paperSubstantial claim1 of 1 sources checked- cites Kwon et al. (2023) source
The “Shared prefix” section says vLLM reserves physical blocks for predefined shared prefixes, with their KV caches stored in advance by the service provider. It does not say these prefixes are inflexible to refresh at scale or that developers and the provider must both participate in an update loop. (LLM, conf=0.83)
Experimental — accuracy not yet measured
- Premisereasoning§ 1 Introduction
The Kwon et al. proposal has memory waste in case of long system prompts and low hit rate.
“ii) there is memory waste in case of long system prompts and low hit rate”
Quote in the paperSubstantial claim1 of 1 sources checked- cites Kwon et al. (2023) source
The “Shared prefix” section (page unavailable) says vLLM reserves physical blocks for predefined cached prefixes, so the proposal does allocate memory to them. It does not discuss low hit rates or say that long system prompts cause memory waste under that condition; the waste described in §3.1 concerns earlier systems instead. (LLM, conf=0.75)
Experimental — accuracy not yet measured
- Findingcitation§ 1 Introduction
System prompts in LLM-based applications can be long (more than 1K tokens).
“we reveal that system prompts can be long (§2.1), providing opportunities for optimizing selfattention”
- Also stated in: § 1 Introduction · plain
“gyudoza (2023) shows that system prompts of various LLM-based applications have more than 1K tokens.”
Quote in the paperSubstantial claim0 of 1 sources checked- cites gyudoza (2023)
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Also stated in: § 1 Introduction · plain
- Findingreasoning§ Abstract
ChunkAttention can detect matching prompt prefixes across multiple requests and share their key/value tensors in memory at runtime to improve KV cache memory utilization.
“we introduce ChunkAttention, a prefix-aware self-attention module that can detect matching prompt prefixes across multiple requests and share their key/value tensors in memory at runtime to improve the memory utilization of KV cache.”
Quote in the paperSubstantial claim - Findingown data§ Abstract
ChunkAttention speeds up the self-attention kernel by 3.2-4.8× compared to the state-of-the-art implementation, for system prompt lengths from 1024 to 4096.
“Experiments show that ChunkAttention can speed up the self-attention kernel by 3.2-4.8 × compared to the state-of-the-art implementation, with the length of the system prompt ranging from 1024 to 4096.”
Quote in the paperSubstantial claim
- Findingown data§ 4.1 Microkernel Evaluation
Sharing KV cache memory physically reduces latency by up to 52% compared to PagedAttn at n_s=4096.
“the hardware cache helps reduce its latency by up to 52% compared to PagedAttn ( n s =4096): repeatedly accessing the same physical memory blocks provides significant performance gain.”
Quote in the paperSubstantial claim- The authors state this as an assumption: Iteration-based batching is enabled to form batches for ChunkAttention's kernel to run efficiently.
- Findingown data§ 4.2 End-to-end Evaluation
ChunkLlama yields the fastest inference speed.
“ChunkLlama yields the fastest inference speed, as shown in Figure 5.”
Quote in the paperSubstantial claim - Findingown data§ 4.1 Microkernel Evaluation
Given n_s=2048, ChunkAttn is 2.0× faster than PagedAttn* when n_c reaches 512.
“However, given n s =2048, ChunkAttn is still 2.0 × (145K against 73K) and 1.5 × (70K against 46K) faster than PagedAttn* when n c reaches 512 and 2048 respectively.”
Quote in the paperSubstantial claim - Findingown data§ 4.1 Microkernel Evaluation
Given n_s=2048, ChunkAttn is 1.5× faster than PagedAttn* when n_c reaches 2048.
“However, given n s =2048, ChunkAttn is still 2.0 × (145K against 73K) and 1.5 × (70K against 46K) faster than PagedAttn* when n c reaches 512 and 2048 respectively.”
Quote in the paperSubstantial claim - Findingown data§ 4.1 Microkernel Evaluation
ChunkAttn outperforms PagedAttn* by 2.8-3.2× for n_s from 1024 to 4096.
“ChunkAttn outperforms PagedAttn* by 2.8-3.2 × , with a range of n s from 1024 to 4096.”
Quote in the paperSubstantial claim- The authors state this as an assumption: Iteration-based batching is enabled to form batches for ChunkAttention's kernel to run efficiently.
- Findingown data§ 4.1 Microkernel Evaluation
Given 2048 shared tokens, the speedup of ChunkAttn over PagedAttn drops to 2.3× when n_c reaches 2048.
“Given 2048 shared tokens, ChunkAttn yields 3.6 × token rate improvement compared to PagedAttn when n c reaches 512, and the speedup drops to 2.3 × when n c reaches 2048.”
Quote in the paperNot a substantial claim - Findingown data§ 4.1 Microkernel Evaluation
TPP does not cause performance regression when no token is shared (n_s=0, ChunkAttn vs. PagedAttn*).
“TPP does not cause performance regression when no token is shared ( n s =0, ChunkAttn vs. PagedAttn* in Table 3).”
Quote in the paperSubstantial claim- The authors state this as an assumption: Iteration-based batching is enabled to form batches for ChunkAttention's kernel to run efficiently.
- Findingown data§ 6 Conclusion
The ChunkAttention kernel can outperform SOTA PagedAttention by 3.2-4.8× with a shared system prompt of 1024 to 4096 tokens on A100 (80G) by applying prefix-aware KV cache and two-phase partition.
“Experiments show that the ChunkAttention kernel can achieve comparable throughput with SOTA PagedAttention kernel without shared system prompts and can outperform it by 3.2-4.8 × with a shared system prompt of 1024 to 4096 tokens on A100 (80G) by applying prefix-aware KV cache and two-phase partition.”
Quote in the paperSubstantial claim - Findingown data§ 4.1 Microkernel Evaluation
ChunkAttn and PagedAttn* outperform Naive, xformers, FlashAttn, and PagedAttn, which are agnostic to shared token count.
“ChunkAttn and PagedAttn* outperform Naive, xformers, FlashAttn, and PagedAttn, which are agnostic to shared token count.”
Quote in the paperSubstantial claim- The authors state this as an assumption: Iteration-based batching is enabled to form batches for ChunkAttention's kernel to run efficiently.
- Findingreasoning§ 4.2 End-to-end Evaluation
The peak batch size is reduced by 20%-40% with ChunkLlama.
“The peak batch size is also reduced by 20%-40% since ChunkLlama can decode faster.”
Quote in the paperNot a substantial claim - Findingown data§ 4.1 Microkernel Evaluation
For all implementations except ChunkAttn and PagedAttn*, throughput peaks when the batch size reaches 16 due to memory-bound.
“For all implementations except ChunkAttn and PagedAttn*, the throughput peaks when the batch size reaches 16 due to memory-bound.”
Quote in the paperNot a substantial claim - Findingown data§ 4.1 Microkernel Evaluation
Given n_s=2048, ChunkAttn's throughput continues to grow from 155K to 224K toks/s for batch sizes ranging from 16 to 96.
“Given n s is 2048, ChunkAttn's throughput continues to grow from 155K to 224K toks/s for the batch size ranging from 16 to 96 due to better data locality and improved arithmetic intensity.”
Quote in the paperNot a substantial claim - Findingown data§ 6 Conclusion
The ChunkAttention kernel can achieve comparable throughput with SOTA PagedAttention kernel without shared system prompts.
“Experiments show that the ChunkAttention kernel can achieve comparable throughput with SOTA PagedAttention kernel without shared system prompts and can outperform it by 3.2-4.8 × with a shared system prompt of 1024 to 4096 tokens on A100 (80G) by applying prefix-aware KV cache and two-phase partition.”
Quote in the paperSubstantial claim - Findingown data§ 4.1 Microkernel Evaluation
Given 2048 shared tokens, ChunkAttn yields a 3.6× token rate improvement compared to PagedAttn when n_c reaches 512.
“Given 2048 shared tokens, ChunkAttn yields 3.6 × token rate improvement compared to PagedAttn when n c reaches 512, and the speedup drops to 2.3 × when n c reaches 2048.”
Quote in the paperSubstantial claim - Findingown data§ 4.2 End-to-end Evaluation
The KV cache memory usage is reduced by 70%-90% with long shared prefixes.
“The KV cache memory usage is reduced by 70%-90% with long shared prefixes.”
Quote in the paperSubstantial claim - Findingreasoning§ 1 Introduction
An inference server with 8*A100 (80G) can only hold 70000 tokens or 35 sequences of 2K context tokens.
“The memory of an inference server with 8*A100 (80G) can only hold 70000 tokens or 35 sequences of 2K context tokens.”
Quote in the paperSubstantial claim - Findingown data§ 4.2 End-to-end Evaluation
No performance regression is observed in ChunkLlama without shared prefix tokens.
“No performance regression is observed in ChunkLlama without shared prefix tokens.”
Quote in the paperSubstantial claim - Interpretationreasoning§ Abstract
Breaking monolithic key/value tensors into smaller chunks and structuring them into an auxiliary prefix tree enables prefix-aware KV cache sharing.
“This is achieved by breaking monolithic key/value tensors into smaller chunks and structuring them into the auxiliary prefix tree.”
Quote in the paperSubstantial claim - Interpretationreasoning§ Abstract
In multi-tenant LLM serving, the compute and memory operation cost of self-attention can be optimized by exploiting shared system prompt prefixes across requests.
“In multi-tenant LLM serving scenarios, the compute and memory operation cost of self-attention can be optimized by using the probability that multiple LLM requests have shared system prompts in prefixes.”
Quote in the paperSubstantial claim - Interpretationreasoning§ Abstract
The two-phase partition algorithm improves data locality during self-attention computation in the presence of shared system prompts.
“we design an efficient self-attention kernel, where a two-phase partition algorithm is implemented to improve the data locality during self-attention computation in the presence of shared system prompts.”
Quote in the paperSubstantial claim- The authors state this as an assumption: Iteration-based batching is enabled to form batches for ChunkAttention's kernel to run efficiently.
- Premisecitation§ 3.2 Two-phase Partition (TPP)
Batching turns the query from a vector into a matrix, allowing efficient matrix multiplications with tensor cores.
“In addition to improved data locality, another advantage of batching is to turn the query from a vector into a matrix, allowing efficient matrix multiplications with tensor cores (Choquette et al., 2021).”
Quote in the paperSubstantial claim0 of 1 sources checked- cites Choquette et al., 2021 source
- Interpretationnone given§ 3.2 Two-phase Partition (TPP)
During prefilling, prefix lookup avoids repeated computation of KV projection and position embedding for matched prompt prefixes.
“During prefilling, we perform a prefix lookup to avoid repeated computation of KV projection and position embedding for matched prompt prefixes.”
Quote in the paperSubstantial claim - Interpretationreasoning§ 3.2 Two-phase Partition (TPP)
The two-phase partition algorithm balances parallelization and cache locality.
“The two-phase partition algorithm balances parallelization and cache locality.”
Quote in the paperSubstantial claim- The authors state this as an assumption: Iteration-based batching is enabled to form batches for ChunkAttention's kernel to run efficiently.
- Interpretationreasoning§ 4.1 Microkernel Evaluation
Since Naive, xformers, and FlashAttn are built on monolithic KV tensors, they cannot be prefix-aware by partially sharing KV cache of prompt prefixes.
“Since Naive, xformers, and FlashAttn are all built on monolithic KV tensors, they cannot be prefix-aware by partially sharing KV cache of prompt prefixes.”
Quote in the paperSubstantial claim - Interpretationreasoning§ 4.2 End-to-end Evaluation
The peak batch size reduction is because ChunkLlama can decode faster.
“The peak batch size is also reduced by 20%-40% since ChunkLlama can decode faster.”
Quote in the paperNot a substantial claim - Premisecitation§ 1 Introduction
The shared system prompt in LLM-based applications leads to redundancy in KV cache.
“On the other hand, the shared system prompt in LLM-based applications leads to redundancy in KV cache (Anthropic, 2023).”
Quote in the paperSubstantial claim0 of 1 sources checked- cites Anthropic, 2023
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Interpretationreasoning§ 3.3 Further Optimizations
The overhead of context generation is amortized.
“The overhead is amortized.”
Quote in the paperSubstantial claim- The authors state this as an assumption: Iteration-based batching is enabled to form batches for ChunkAttention's kernel to run efficiently.
- Interpretationown data§ 6 Conclusion
ChunkAttention is a novel self-attention module to efficiently manage KV cache and accelerate the self-attention kernel for LLMs inference.
“In this paper, we propose ChunkAttention, a novel self-attention module, to efficiently manage KV cache and accelerate the self-attention kernel for LLMs inference.”
Quote in the paperSubstantial claim - Premisenone given§ 4.1 Microkernel Evaluation
PagedAttn does not implement PAKV.
“PagedAttn does not implement PAKV either.”
Quote in the paperNot a substantial claim - Interpretationreasoning§ 3.1 Prefix Aware KV Cache (PAKV)
Slicing the query tensor in self-attention is particularly efficient during kernel computation because of the contiguity property.
“Therefore, slicing the query tensor in self-attention is particularly efficient during kernel computation, which will be discussed in more detail in the next section.”
Quote in the paperSubstantial claim - Interpretationreasoning§ 5 Related Work
ChunkAttention and FlashAttention are complementary.
“As a result, our work and FlashAttention are complementary.”
Quote in the paperNot a substantial claim - Premisereasoning§ 4.1 Microkernel Evaluation
None of the kernels support the TPP algorithm.
“None of the kernels support the TPP algorithm.”
Quote in the paperNot a substantial claim - Interpretationown data§ 4.1 Microkernel Evaluation
ChunkAttn's throughput growth from batch size 16 to 96 is due to better data locality and improved arithmetic intensity.
“Given n s is 2048, ChunkAttn's throughput continues to grow from 155K to 224K toks/s for the batch size ranging from 16 to 96 due to better data locality and improved arithmetic intensity.”
Quote in the paperSubstantial claim - Premisecitation§ 7 Limitations
Performance is often lowest when models must use information in the middle of long input contexts.
“In particular, performance is often lowest when models must use information in the middle of long input contexts.”
Quote in the paperSubstantial claim0 of 1 sources checked- cites Liu et al. (2023)
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Interpretationnone given§ 7 Limitations
To share key/value tensors in memory, the shared system prompt must appear at the beginning of the sequence.
“To share the key/value tensors in memory, the shared system prompt must appear at the beginning of the sequence.”
Quote in the paperSubstantial claim - Premisecitation§ 7 Limitations
Language model performance degrades significantly when changing the position of relevant information.
“Liu et al. (2023) reveals that language model performance degrades significantly when changing the position of relevant information, indicating that models struggle to access and use information in long input contexts robustly.”
Quote in the paperSubstantial claim0 of 1 sources checked- cites Liu et al. (2023)
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Interpretationnone given§ 1 Introduction
The memory complexity of self-attention grows linearly with context length.
“The memory complexity grows linearly with context length.”
Quote in the paperNot a substantial claim - Interpretationreasoning§ 3.1 Prefix Aware KV Cache (PAKV)
By sharing common prefixes, the number of sequences that can be processed simultaneously is increased by approximately 1/(1-r).
“By sharing common prefixes, the number of sequences that can be processed simultaneously is increased by approximately 1 / (1 -r ) .”
Quote in the paperNot a substantial claim - Interpretationreasoning§ 6 Conclusion
ChunkAttention addresses the challenge of detecting and removing redundant KV cache at runtime.
“It addresses the challenge of detecting and removing redundant KV cache at runtime.”
Quote in the paperSubstantial claim - Premisereasoning§ 1 Introduction
No work has been done to optimize the self-attention kernel in the presence of shared system prompts.
“iii) no work has been done to optimize the self-attention kernel in the presence of shared system prompts.”
Quote in the paperNot a substantial claim - Interpretationreasoning§ 5 Related Work
ChunkAttention proposes a two-phase partition algorithm to explore optimization opportunities shared system prompts bring to the self-attention kernel.
“Based on the context captured in the prefix tree, our work further proposes a twophase partition algorithm to explore the optimization opportunities shared system prompts bring to the self-attention kernel, which is another difference between our work and the existing work.”
Quote in the paperSubstantial claim - Interpretationnone given§ 6 Conclusion
ChunkAttention successfully adopts the prefix tree to create a prefix-aware KV cache.
“We successfully adopt the prefix tree to create a prefix-aware KV cache.”
Quote in the paperSubstantial claim
- It may state two separate claims as one. 26
- Probably not a substantial claim. 17
It may state two separate claims as one.
Premisecitation§ 1 IntroductionThe only related work is Kwon et al. (2023), which reserves memory for key/value tensors of predefined system prompts.
“To our knowledge, the only related work is a proposal by Kwon et al. (2023), in which the service provider reserves memory for key/value tensors of a set of predefined system prompts from application developers.”
Quote in the paperSubstantial claim1 of 1 sources checked- cites Kwon et al. (2023) source
The “Shared prefix” section says the LLM service provider can reserve physical blocks for predefined shared prefixes and store their KV cache in advance. It does not identify those prefixes specifically as system prompts supplied by application developers. The sentence’s “only related work” framing is the citing authors’ own assessment, not a claim established by this source. (LLM, conf=0.84)
Experimental — accuracy not yet measured
Probably not a substantial claim. The paper’s conclusions may not depend on it.
Premisecitation§ 5 Related WorkThe most relevant work on optimizing the memory utilization of KV cache is PagedAttention in vLLM.
“The most relevant work on optimizing the memory utilization of KV cache is PagedAttention in vLLM (Kwon et al., 2023).”
Quote in the paperNot a substantial claim1 of 1 sources checked- cites Kwon et al., 2023 source
It may state two separate claims as one.
Premisecitation§ 5 Related WorkOnly a proposal on service providers to pre-configure shared prompts is mentioned in vLLM, and it is not implemented in vLLM releases up to 0.2.7.
“However, only a proposal on service providers to pre-configure shared prompts is mentioned, and it is not implemented in vLLM releases (up to 0.2.7).”
Quote in the paperNot a substantial claim1 of 1 sources checked- cites Kwon et al., 2023 source
The “Shared prefix” section describes how a service provider can predefine shared prefixes and reserve KV-cache blocks for them. Section 6.4 also reports vLLM performance tests using shared prefixes, so the paper does more than merely mention a proposal. Our copy does not discuss vLLM releases through 0.2.7 or establish whether those releases implemented the feature. (LLM, conf=0.87)
Experimental — accuracy not yet measured
It may state two separate claims as one.
Premisecitation§ 5 Related WorkPartition strategies in ChunkAttention are built on online softmax and inspired by FlashAttention.
“Partition strategies in ChunkAttention are built on online softmax (Milakov and Gimelshein, 2018) and inspired by FlashAttention (Dao et al., 2022; Dao, 2023), which adopted the same algorithm.”
Quote in the paperNot a substantial claim3 of 3 sources checkedIt may state two separate claims as one.
Findingcitation§ 1 IntroductionThe self-attention module has low arithmetic intensity and high latency.
“The selfattention module has low arithmetic intensity (Williams et al., 2009) and high latency.”
Quote in the paperSubstantial claim1 of 1 sources checked- cites Williams et al., 2009 source
The citation is attached to the claim that self-attention has low arithmetic intensity, not to the separate statement about latency. Section 3, “The Roofline Model,” defines intensity metrics, and Section 6 demonstrates the model on other kernels. Searches for self-attention and transformer-related terms found no passage assessing the arithmetic intensity of a self-attention module. (LLM, conf=0.75)
Experimental — accuracy not yet measured
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
Premisecitation§ 5 Related WorkFlashAttention-2 altered tiling strategies and additionally doubled the speed.
“FlashAttention-2 altered tiling strategies and additionally doubled the speed.”
Quote in the paperNot a substantial claim1 of 1 sources checked- cites Dao, 2023 source
Probably not a substantial claim. The paper’s conclusions may not depend on it.
Premisecitation§ 5 Related WorkPagedAttention introduces the paging technique in OS to solve the problem of memory waste caused by dynamic and unknown sequence lengths during decoding.
“It introduces the paging technique in OS to solve the problem of memory waste caused by dynamic and unknown sequence lengths during decoding.”
Quote in the paperNot a substantial claim1 of 1 sources checked- cites Kwon et al., 2023 source
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
Premisecitation§ 5 Related WorkFlashAttention accelerated self-attention by 2-4× while cutting memory operations by 10-20×.
“FlashAttention thoroughly researched and implemented various tiling techniques, accelerating selfattention by 2-4 × while cutting memory operations by 10-20 × .”
Quote in the paperNot a substantial claim1 of 1 sources checked- cites Dao et al., 2022 source
Appendix B.5 (p. 22) says FlashAttention is 2–4× faster than standard attention. The Introduction (p. 2) reports up to 9× fewer HBM accesses, not 10–20× fewer memory operations; the up-to-20× figure in §4.3 (p. 10) concerns memory footprint instead. (LLM, conf=0.88)
Experimental — accuracy not yet measured
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it.
Premisecitation§ 7 LimitationsFine-tuning is another promising way to infuse domain knowledge into LLMs.
“In addition to using system prompts, fine-tuning is another promising way to infuse domain knowledge into LLMs (Houlsby et al., 2019; Hu et al., 2023).”
Quote in the paperNot a substantial claim1 of 2 sources checked- cites Houlsby et al., 2019 source
The Introduction (p. 1) describes fine-tuning a pretrained network for a downstream task and says it often performs better than feature-based transfer. Related Work (p. 8) also reports strong results from fine-tuning language models. These passages support fine-tuning as a promising NLP transfer method, but they do not address infusing domain knowledge into LLMs or compare fine-tuning with system prompts. (LLM, conf=0.83)
Experimental — accuracy not yet measured - cites Hu et al., 2023
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
Findingown data§ 1 IntroductionFor a ChatGPT-like online chatbot, the system prompt can be as long as 1766 tokens with only 6 plugins activated, and all requests sent to the chatbot share one single system prompt.
“For a ChatGPT-like online chatbot, the system prompt can be as long as 1766 tokens with only 6 plugins activated, and all requests sent to the chatbot share one single system prompt (Appendix A).”
Quote in the paperNot a substantial claimProbably not a substantial claim. The paper may give no reason why it holds.
Findingnone given§ AbstractSelf-attention is a significant source of inference latency for long sequences in LLMs.
“Self-attention is an essential component of large language models (LLM) but a significant source of inference latency for long sequences.”
Quote in the paperNot a substantial claimProbably not a substantial claim. The paper may give no reason why it holds.
Findingnone given§ 1 IntroductionUsing FP16, the KV cache for each token in GPT3(175B) requires 4.5MB of memory.
“For instance, using FP16, the KV cache for each token in GPT3(175B) requires 4.5MB of memory.”
Quote in the paperNot a substantial claimIt may state two separate claims as one.
Findingown data§ 1 IntroductionChunkAttention can be significantly faster as the length of shared system prompts grows and has no performance degradation without shared system prompts, compared to existing highly optimized implementations.
“Our experiments show that ChunkAttention can be significantly faster as the length of shared system prompts grows and has no performance degradation without shared system prompts, compared to existing highly optimized implementations.”
Quote in the paperSubstantial claimIt may state two separate claims as one.
Findingown data§ 4.1 Microkernel EvaluationAs decoding proceeds, sequences start to diverge, and the performance gain of ChunkAttn gradually decreases.
“As the decoding proceeds, sequences start to diverge, and the performance gain of ChunkAttn gradually decreases, as shown in Figure 3.”
Quote in the paperNot a substantial claimProbably not a substantial claim. The paper may give no reason why it holds. It may state two separate claims as one.
Findingnone given§ 2.1 Shared System PromptThe system prompt is shared between multiple requests and can be very long.
“The system prompt is shared between multiple requests and can be very long. This can be observed in various LLM-based applications, from online chatbots to offline experiments.”
Quote in the paperNot a substantial claimIt may state two separate claims as one.
Findingown data§ 4.1 Microkernel EvaluationNaive is 6.6× and 2.1× slower than ChunkAttn and PagedAttn*, respectively, at n_s=4096.
“The Naive is 6.6 × and 2.1 × slower than ChunkAttn and PagedAttn*, respectively ( n s =4096).”
Quote in the paperSubstantial claim- The authors state this as an assumption: Iteration-based batching is enabled to form batches for ChunkAttention's kernel to run efficiently.
It may state two separate claims as one.
Interpretationreasoning§ 3.2 Two-phase Partition (TPP)Without the partial attention results generated by the chunk-first phase, the sequence-first phase needs to load shared chunks from RAM b times, which adds significant MOPs.
“However, without the partial attention results generated by the chunk-first phase, it needs to load shared chunks from RAM b times, which adds significant MOPs.”
Quote in the paperSubstantial claim- The authors state this as an assumption: Iteration-based batching is enabled to form batches for ChunkAttention's kernel to run efficiently.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. It may state two separate claims as one.
Interpretationnone given§ 3.3 Further OptimizationsOn CPU devices, the overhead of serializing is insignificant, and reduction can be implemented using spin locks.
“However, on CPU devices, the overhead of serializing is insignificant, and reduction can be implemented using spin locks.”
Quote in the paperNot a substantial claim- The authors state this as an assumption: Iteration-based batching is enabled to form batches for ChunkAttention's kernel to run efficiently.
It may state two separate claims as one.
Interpretationreasoning§ 7 LimitationsWhen application developers do not put the system prompt at the beginning, KV caches of the entire sequences are different, and PAKV cannot save memory in this case, although they have a large number of tokens in common.
“As a result, when application developers do not put the system prompt at the beginning for performance concerns after evaluations or unintentional mistakes, KV caches of the entire sequences are different, and PAKV cannot save memory in this case, although they have a large number of tokens in common.”
Quote in the paperSubstantial claimIt may state two separate claims as one.
Interpretationreasoning§ 3.1 Prefix Aware KV Cache (PAKV)In memory-limited inference scenarios, sharing common prefixes helps increase batch size and thus improve throughput.
“In memory-limited inference scenarios, this helps increase the batch size and thus improve throughput.”
Quote in the paperSubstantial claimProbably not a substantial claim.
Interpretationreasoning§ 3.1 Prefix Aware KV Cache (PAKV)The KV cache should be made prefix-aware by organizing the KV cache of all sequences under decoding into a prefix tree.
“Given the property, we argue that the KV cache should be made prefix-aware, which is to organize the KV cache of all sequences under decoding into a prefix tree.”
Quote in the paperSubstantial claimIt may state two separate claims as one.
Interpretationreasoning§ 1 IntroductionChunkAttention's KV cache only stores key/value tensors of sequences currently in decoding and has zero memory waste.
“The KV cache only stores key/value tensors of sequences currently in decoding and has zero memory waste.”
Quote in the paperSubstantial claimIt may state two separate claims as one.
Study design and conductown data§ 6 ConclusionChunkAttention was evaluated in various configurations and at different levels, proving its feasibility and that side effects can be managed.
“We evaluate ChunkAttention in various configurations and at different levels, proving its feasibility and the side effects can be managed.”
Quote in the paperSubstantial claimIt may state two separate claims as one.
Interpretationreasoning§ 3.1 Prefix Aware KV Cache (PAKV)When multiple sequences share common prefix tokens, key/value tensors are the same and thus can be shared in memory.
“When multiple sequences share common prefix tokens, key/value tensors are the same and thus can be shared in memory.”
Quote in the paperNot a substantial claimIt may state two separate claims as one.
Interpretationreasoning§ 1 IntroductionKV cache restricts batch size and system throughput.
“KV cache additionally restricts the batch size and system throughput.”
Quote in the paperSubstantial claimIt may state two separate claims as one.
Interpretationreasoning§ 3.3 Further OptimizationsChunkAttention manages context generation overhead by latency hiding and lazy context copy.
“ChunkAttention manages the overhead in two ways: i) latency hiding. The context generation step on CPU can be overlapped with other kernels on GPU prior to self-attention. ii) lazy context copy.”
Quote in the paperNot a substantial claim- The authors state this as an assumption: Iteration-based batching is enabled to form batches for ChunkAttention's kernel to run efficiently.
Probably not a substantial claim. The paper may give no reason why it holds. It may state two separate claims as one.
Premisenone given§ 5 Related WorkFlashAttention is inflexible regarding noncontiguous memory or variable sequence lengths, making it more suitable for model training than inference.
“However, FlashAttention is inflexible regarding noncontiguous memory or variable sequence lengths, making it more suitable for model training than inference.”
Quote in the paperNot a substantial claimIt may state two separate claims as one.
Interpretationreasoning§ 5 Related WorkChunkAttention handles variable sequence lengths during decoding and batches attention operations of several sequences to reduce memory operations.
“ChunkAttention handles variable sequence lengths during decoding and batches attention operations of several sequences to reduce memory operations.”
Quote in the paperSubstantial claimIt may state two separate claims as one.
Interpretationreasoning§ 4.1 Microkernel EvaluationThe improvement of ChunkAttn over PagedAttn* is lower because PagedAttn* benefits from physically shared KV cache memory, and only TPP makes a difference.
“The improvement of ChunkAttn over PagedAttn* is lower since PagedAttn* benefits from physically shared KV cache memory, and only TPP makes a difference here.”
Quote in the paperSubstantial claimIt may state two separate claims as one.
Interpretationreasoning§ 1 IntroductionChunkAttention's KV cache is a prefix tree built with chunked context tokens and key/value tensors, making it prefix-aware and able to dynamically detect and remove redundancy at runtime without human involvement.
“KV cache in ChunkAttention is a prefix tree built with chunked context tokens and key/value tensors. Thus, the KV cache is prefixaware and can dynamically detect and remove redundancy at runtime without human involvement.”
Quote in the paperSubstantial claimProbably not a substantial claim. The paper’s conclusions may not depend on it.
Interpretationreasoning§ 7 LimitationsIt is not cost-efficient for each application to fine-tune models and deploy private instances.
“It is not cost-efficient for each application to finetune models and deploy private instances.”
Quote in the paperNot a substantial claim- Rests on an assumption: LLMs are typically pre-trained and centrally hosted for multiple applications to share due to high training and deployment cost.
Probably not a substantial claim. The paper may give no reason why it holds. It may state two separate claims as one.
Interpretationnone given§ 5 Related WorkChunkAttention uses a prefix tree to manage memory and aims to discover redundancy in KV cache across user requests at runtime automatically.
“Our solution, which differs from the paging one, uses the prefix tree to manage memory and aims to discover redundancy in KV cache across user requests at runtime automatically.”
Quote in the paperNot a substantial claimProbably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
Premisenone given§ 5 Related WorkThere is little gain when the query token count is always one during decoding.
“There is little gain when the query token count is always one during decoding.”
Quote in the paperNot a substantial claimProbably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
Study design and conductreasoning§ 4.1 Microkernel EvaluationPagedAttn* simulates the KV cache sharing scenario and helps observe the performance of PagedAttn's CUDA kernel.
“However, its paging design enables us to manually create a fixed page table, mapping virtually non-shared memory to the same physical memory. It simulates the KV cache sharing scenario and helps us observe the performance of PagedAttn's CUDA kernel, which is denoted as PagedAttn*.”
Quote in the paperNot a substantial claimProbably not a substantial claim. The paper may give no reason why it holds.
Interpretationnone given§ 3.1 Prefix Aware KV Cache (PAKV)A key property of the prefix tree is that sequences covered by each chunk in the prefix tree are contiguous in the sequence index dimension.
“A key property of the prefix tree is that sequences covered by each chunk in the prefix tree are contiguous in the sequence index dimension.”
Quote in the paperNot a substantial claim
The authors state this as an assumption.
Iteration-based batching is enabled to form batches for ChunkAttention's kernel to run efficiently.
“The ChunkAttention in this paper assumes that iteration-based batching is enabled to form batches for its kernel to run efficiently.”
The paper relies on this without support.
LLMs are typically pre-trained and centrally hosted for multiple applications to share due to high training and deployment cost.
“Due to the high training and deployment cost, LLMs are typically pre-trained and centrally hosted for multiple applications to share.”
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
vLLM and ChunkedAttention explore some simple reuse cases (e.g., system prompt sharing) but do not cover multi-level tree-structured sharing or LRU caching.
“vLLM [23] and ChunkedAttention [58] explore some simple reuse cases (e.g., system prompt sharing) but do not cover multi-level tree-structured sharing or LRU caching.”
From SGLang: Efficient Execution of Structured Language Model Programs (2023) · cites this paper as [58]
ChunkAttention does study shared system prompts, but the claim that it does not cover multi-level tree-structured sharing conflicts with §3.1, which describes shared KV-cache chunks in a prefix tree with parent–child relationships. The Introduction also says the cache stores tensors only for sequences currently decoding; this is consistent with not retaining them through LRU caching, but does not rescue the tree-sharing claim. (LLM, conf=0.87)
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredHydraGen, FlashInfer, and ChunkedAttention focus on CUDA kernel optimizations and do not include the concept of an LRU cache.
“HydraGen [18], FlashInfer [59], and ChunkedAttention [58] focus on CUDA kernel optimizations and do not include the concept of an LRU cache.”
From SGLang: Efficient Execution of Structured Language Model Programs (2023) · cites this paper as [58]
The Abstract and Conclusion describe an optimized self-attention kernel, supporting the kernel-optimization part of the claim. But they also present prefix-aware KV-cache management as a central contribution, so describing ChunkAttention as focused on kernel optimization alone is incomplete. Section 3.1 describes a pool allocator with used and free chunk lists, not an LRU policy; the copy does not expressly state that LRU is absent. (LLM, conf=0.82)
Experimental — accuracy not yet measuredA significant portion of input prefixes in LLM inference are reusable.
“Opportunities for Reusing KV Caches: A significant portion of input prefixes in LLM inference are reusable [66], presenting an opportunity to reduce redundant computation through KV cache reuse.”
From HotPrefix: Hotness-Aware KV Cache Scheduling for Efficient Prefix Sharing in LLM Inference Systems (2025) · cites this paper as [66]
In ChatGPT-like applications, a single system prompt can exceed 1,700 tokens when multiple plugins are activated, and all requests share this same prompt.
“In ChatGPTlike applications, a single system prompt can exceed 1,700 tokens when multiple plugins are activated [66], and all requests share this same prompt.”
From HotPrefix: Hotness-Aware KV Cache Scheduling for Efficient Prefix Sharing in LLM Inference Systems (2025) · cites this paper as [66]
Many prompts in LLM inference share common prefixes.
“Since many prompts share common prefixes, prefix sharing has been proposed to reuse shared prefix KV caches during inference.”
From HotPrefix: Hotness-Aware KV Cache Scheduling for Efficient Prefix Sharing in LLM Inference Systems (2025) · cites this paper as [66]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
ChunkAttention said to have zero memory waste, but elsewhere memory loss is bounded not zero
MinorOne passage says the KV cache has zero memory waste, while another states that some memory space for alignment is unused and the memory loss is bounded by a nonzero expression. These claims cannot both be true as written.
“The KV cache only stores key/value tensors of sequences currently in decoding and has zero memory waste.”
“Some memory space for alignment is unused. Given that the sequence length is n , the memory loss is bounded by ( c -1) /n .”
Citations
Cites
The publisher hasn’t deposited this paper’s reference list.
Research works printed but not linked: 1
- Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAI blog , 1(8):9.
Cited by
Sorted by citations
RAGCache: Efficient Knowledge Caching for Retrieval-Augmented Generation
Jin, C., Zhang, Z., Jiang, X., Liu, F., Liu, X., Liu, X., & Jin, X. (2024) · arXiv · doi:10.48550/arxiv.2404.12457 · Cited by 61
RAGCache: Efficient Knowledge Caching for Retrieval-Augmented Generation
Jin, C., Zhang, Z., Jiang, X., Liu, F., Liu, S., Liu, X., & Jin, X. (2026) · ACM Transactions on Computer Systems · doi:10.1145/3768628 · Cited by 61
Belkhouribchia, J., & Pen, J. J. (2025) · Frontiers in Nutrition · doi:10.3389/fnut.2025.1635682 · Cited by 43
“This mechanism is what allows LLMs to interpret nuanced queries and maintain contextual coherence over long passages of text (24–26).”
SGLang: Efficient Execution of Structured Language Model Programs
Zheng, L., Yin, L., Xie, Z., Sun, C., Huang, J., Yu, C. H., Cao, S., Kozyrakis, C., Stoica, I., Gonzalez, J. E., et al. (2023) · arXiv · doi:10.48550/arxiv.2312.07104 · Cited by 145
“…the first solution that supports multi-level sharing, cache-aware scheduling, frontend-runtime co-scheduling, and distributed cases. vLLM [23] and ChunkedAttention [58] explore some simple reuse cases (e.g., system prompt sharing) but do not cover multi-level tree-structured sharing or LRU caching.”
HeterRAG: Heterogeneous Processing-in-Memory Acceleration for Retrieval-augmented Generation
Liu, C., Liu, H., Chen, D., Huang, Y., Zhang, Y., Xiao, W., Liao, X., & Jin, H. (2025) · Proceedings - International Symposium on Computer Architecture/Proceedings · doi:10.1145/3695053.3731089 · Cited by 16
Self-citationHotPrefix: Hotness-Aware KV Cache Scheduling for Efficient Prefix Sharing in LLM Inference Systems
Li, Y., Gu, R., Huan, C., Wang, Z., Yao, R., Tian, C., & Chen, G. (2025) · Proceedings of the ACM on Management of Data · doi:10.1145/3749168 · Cited by 17
Self-citationAccelerating LLM Serving for Multi-turn Dialogues with Efficient Resource Management
Jeong, J., & Ahn, J. (2025) · International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) · doi:10.1145/3676641.3716245 · Cited by 16
Chen, Y., Guan, W., Azzam, R., & Chen, S. (2025) · AI · doi:10.3390/ai6060127 · Cited by 5
Minimizing Response Latency in LLM-Based Agent Systems: A Comprehensive Survey
Park, G., Lee, S., & Park, Y. (2026) · IEEE Access · doi:10.1109/access.2026.3664226 · Cited by 4
Serving LLM in Distributed GPU Cluster With Fine-Grain Pipeline Constraints
Lin, Y., Peng, S., Wu, S., Li, Y., Lu, C., Ye, K., & Xu, C. (2025) · IEEE Transactions on Services Computing · doi:10.1109/tsc.2025.3603576 · Cited by 3
Self-citationFrom Human Memory to AI Memory: A Survey on Memory Mechanisms in the Era of LLMs
Wu, Y., Liang, S., Zhang, C., Wang, Y., Zhang, Y., Guo, H., Tang, R., & Liu, Y. (2025) · arXiv · doi:10.48550/arxiv.2504.15965 · Cited by 3
“…AdaServe [125], MPIC [126], IntelLLM [127] KV Reuse KV Cache [128], Prompt Cache [83], Contextual Retrieval [84], CacheGen [129], ChunkAttention [130], RAGCache [131], SGLang [132], Ada-KV [133], HCache [134], Cake [135], EPIC [136], RelayAttention [137], Marconi [138], IKS [139], FastCache…”
Keep the Cost Down: A Review on Methods to Optimize LLM' s KV-Cache Consumption
Shi, L., Zhang, H., Yao, Y., Li, Z., & Zhao, H. (2024) · arXiv · doi:10.48550/arxiv.2407.18003 · Cited by 2
PrefixQuant: Eliminating Outliers by Prefixed Tokens for Large Language Models Quantization
Chen, M., Liu, Y., Wang, J., Bin, Y., Shao, W., & Luo, P. (2026) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2026.3711802 · Cited by 33
PEEK: Predictive Queue-Informed KV Cache Management for LLM Serving
Xie, B., Wang, Z., Tanaka, M., & Zheng, Z. (2026) · arXiv · doi:10.48550/arxiv.2607.02525
“…PromptCache [Gim et al., 2024] (module-level KV via positional re-encoding), RAGCache [Jin et al., 2024] (hierarchical RAG cache), ChunkAttention [Ye et al., 2024] (attention over a prefix tree), Hydragen [Juravsky et al., 2024] (fused shared-prefix kernel), CachedAttention [Gao et al., 2024]…”
SimLayerKV: A Simple Framework for Layer-Level KV Cache ReductionPreprint
Zhang, X., Du, C., Du, C., Pang, T., Gao, W., & Lin, M. C. (2024) · Qeios · doi:10.32388/epai4t · Cited by 2
Cache Partitioning for Mitigating Timing Side-Channel Attacks in LLM Serving Systems
Pang, Z., Wang, W., & Liao, Y. (2024) · IEEE International Conference on Frontiers Technology of Information and Computer (ICFTIC) · doi:10.1109/icftic64248.2024.10913329
Accelerating Multi‑turn LLM Agent Workflows via Context Templating and Opportunistic Prefill
Wang, H., Wen, Y., & Zhang, W. (2025) · Lecture notes in computer science · doi:10.1007/978-981-96-9901-8_14
EFIM: Efficient Serving of LLMs for Infilling Tasks with Improved KV Cache Reuse
Guo, T., Dong, H., Leng, Y., Liu, F., Lin, C., Xiao, N., & Zhang, X. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-99857-7_24
ContextCache: Task-Aware Lifecycle Management for Memory-Efficient LLM Agent Deployment
Liu, T., Guo, P., Feng, D., & Wang, P. (2025) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros60139.2025.11247595
XY-Serve: End-to-End Versatile Production Serving for Dynamic LLM Workloads
Song, M., Tang, X., Hou, F., Li, J., Wei, W., Ma, Y., Xiao, R., Si, H., Jiang, D., Yin, S., et al. (2025) · International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) · doi:10.1145/3760250.3762228
DKC-LLM: Dynamic Knowledge Caching for Large Language Models in Business Applications
Khaliq, A., & Adebayo, K. J. (2026) · IEEE Access · doi:10.1109/access.2026.3662344
REPA: Re configurable P IM for the Joint A cceleration of KV Cache Offloading and Processing
Hong, Y., Yang, J., Peng, B., & Yao, J. (2026) · International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) · doi:10.1145/3779212.3790212
Accelerating Language Model Workflows with Prompt Choreography
Bai, T., & Eisner, J. M. (2026) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl.a.643
Accelerating Local LLMs on Resource-Constrained Edge Devices via Distributed Prompt Caching
Matsutani, H., Matsuda, N., & Sugiura, N. (2026) · Workshop on Machine Learning and Systems (EuroMLSys) · doi:10.1145/3805621.3807647
CALM: Cost-efficient, model Agnostic, Low-latency Modular LLM Deployment framework
Bansal, A., Gill, D. K., Joshi, R., Pham, H., Zamparo, L., & Liu, W. (2026) · International Conference on Advances in Artificial Intelligence and Machine Learning (AAIML) · doi:10.1109/aaiml67890.2026.11498121
Efficient LLM Serving for Agentic Workflows: A Data Systems Perspective
Wadlom, N., Shen, J., & Lu, Y. (2026) · Proceedings of the ACM on Management of Data · doi:10.1145/3802046
CoDec: Prefix-Shared Decoding Kernel for LLMs
Wang, Z., Ning, R., Fang, C., Zhang, Z. J., Lin, X., Ma, S., Zhou, M., Li, X., Wang, Z., Huan, C., et al. (2026) · Proceedings of the ACM on Management of Data · doi:10.1145/3802028 · Cited by 7
ACES: A Chiplet Architecture with Resource Partition and Dynamic Scheduling for Agentic LLMs
Li, H., Li, M., Zhu, Z., & Jia, T. (2026) · Design, Automation, and Test in Europe · doi:10.23919/date69613.2026.11539693
Cache-Enhanced RAG and Graph-RAG for O-RAN
Bano, S., Nezami, Z., Hafeez, M., Zaidi, S. A., & Ahmed, Q. (2025) · IEEE Globecom Workshops (GC Wkshps) · doi:10.1109/gcwkshps68340.2025.11591230
Fine-Grained Energy Accounting in Production LLM Serving
Yuan, X., Liao, H., Zhang, K., Guo, D., & Tang, G. (2026) · Asia-Pacific Workshop on Networking (APNet) · doi:10.1145/3820441.3820447
Path-Compressed Trie for Efficient KV Cache Reuse
Wang, B., & Xu, Y. (2026) · International Conference on Computer and Communication Systems (ICCCS) · doi:10.1109/icccs69761.2026.11613300
Efficient KV Cache Migration for Geo-Distributed LLM Inference in Collaborative Edge Computing
Zhang, M., Cao, J., Wu, T., Chen, X., & Wang, N. (2026) · IEEE International Conference on Distributed Computing Systems (ICDCS) · doi:10.1109/2575-8411.2026.00033
ConServe: Contiguity-Preserving Memory Management for Multi-Turn LLM Serving
Li, B. (2026) · Proceedings - International Symposium on Computer Architecture/Proceedings · doi:10.1109/isca66397.2026.00075
KV Cache Reuse for Elastic LLM Inference on Edge Devices
Wang, P., Zheng, Z., Huang, X., Wu, J., Wu, F., & Chen, G. (2026) · IEEE International Conference on Distributed Computing Systems (ICDCS) · doi:10.1109/2575-8411.2026.00035
Structure Is All You Need to Reuse: Accelerating GraphRAG via Meta-Structure-Aware KV Caching
Luo, R., Gu, C., Yang, J., Liang, H., Gao, Y., Wang, X., He, Q., Wu, S., Yang, L. T., & Jin, H. (2026) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3770855.3817942
InputSnatch: Stealing Input in LLM Services via Cache-Sharing Timing Side-Channel Attacks
Zheng, X., Han, H., Shi, S., Fang, Q., Du, Z., Hu, X., & Guo, Q. (2026) · Lecture notes in computer science · doi:10.1007/978-981-92-4805-6_32
Pegasus: Accelerating Large Language Model Inference with Stateful Prefix Caching
Chen, F., Li, P., Yu, D., & Guo, S. (2026) · IEEE International Conference on Joint Cloud Computing (JCC) · doi:10.1109/jcc72984.2026.00017
Gao, L., Han, S., Wang, X., Wang, X., Zou, Z., Jiang, L., & Zhang, J. (2026) · Intelligent and Converged Networks · doi:10.23919/icn.2026.0008
ALPS: An Application-Aware, Length-Predictive Scheduler for Efficient LLM Inference at Edge
Chen, C., Jiao, L., & Mortier, R. (2026) · IEEE Transactions on Parallel and Distributed Systems · doi:10.1109/tpds.2026.3741399
Anchor-NDP: An importance-based prefix KV management framework for CXL-enabled NDP
Kim, H., Kim, W., & Han, T. H. (2026) · Journal of Systems Architecture · doi:10.1016/j.sysarc.2026.104056
Metadata
- Authors
- Ye, L., Tao, Z., Huang, Y., & Li, Y.
- Affiliations
- Microsoft (United States) (US)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2024
- Pages
- 11608-11620
- ISSN
- 0736-587X (ISSN-L)
- arXiv
- 2402.15220
- OpenAlex
- W4402671542
- Cited by
- 34
- References
- 0
- Citations by year
- 2026: 21 · 2025: 11 · 2024: 2
- FWCI
- 10.02
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Proceedings of the conference - Association for Computational Linguistics. Meeting (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Hardware and Architecture › Parallel Computing and Optimization Techniques
- Keywords
- key-value cache, prefix tree, data locality, inference latency, memory efficiency, large language models
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-01, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-01 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
