Hydragen: High-Throughput LLM Inference with Shared Prefixes
Open accessAbstract
Transformer-based large language models (LLMs) are now deployed to hundreds of millions of users. LLM inference is commonly performed on batches of sequences that share a prefix, such as few-shot examples or a chatbot system prompt. Decoding in this large-batch setting can be bottlenecked by the attention operation, which reads large key-value (KV) caches from memory and computes inefficient matrix-vector products for every sequence in the batch. In this work, we introduce Hydragen, a hardware-aware exact implementation of attention with shared prefixes. Hydragen computes attention over the shared prefix and unique suffixes separately. This decomposition enables efficient prefix attention by batching queries together across sequences, reducing redundant memory reads and enabling the use of hardware-friendly matrix multiplications. Our method can improve end-to-end CodeLlama-13b throughput by up to 32x against competitive baselines, with speedup growing with the batch size and shared prefix length. Hydragen also enables the use of very long shared contexts: with a large batch size, increasing the prefix length from 1K to 16K tokens decreases Hydragen throughput by less than 15%, while the throughput of baselines drops by over 90%. Hydragen generalizes beyond simple prefix-suffix decomposition and can be applied to tree-based prompt sharing patterns, allowing us to further reduce inference time on competitive programming problems by 55%.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
HydraGen, FlashInfer, and ChunkedAttention focus on CUDA kernel optimizations and do not include the concept of an LRU cache.
“HydraGen [18], FlashInfer [59], and ChunkedAttention [58] focus on CUDA kernel optimizations and do not include the concept of an LRU cache.”
From SGLang: Efficient Execution of Structured Language Model Programs (2023) · cites this paper as [18]
Section 3.5 (Implementation; page unavailable) says Hydragen uses no custom CUDA code and is written in PyTorch with existing attention primitives, contrary to the claim that it focuses on CUDA kernel optimizations. The Introduction describes its contribution as decomposing attention over shared prefixes. The paper does not establish what FlashInfer or ChunkedAttention include, and its silence on LRU caching does not verify their exclusion of it. (LLM, conf=0.86)
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredHydragen proposes an efficient implementation of the attention operation for shared prefixes, which is orthogonal to Preble.
“Hydragen Juravsky et al. (2024) is another recent work that proposes an efficient implementation of the attention operation for shared prefixes, which is orthogonal to Preble, as Preble can support any underlying attention kernels.”
From Preble: Efficient Distributed Prompt Scheduling for LLM Serving (2024) · cites this paper as Juravsky et al. (2024)
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
The publisher hasn’t deposited this paper’s reference list.
Research works printed but not linked: 7
- Reza Yazdani Aminabadi, Samyam Rajbhandari, Ammar Ahmad Awan, Cheng Li, Du Li, Elton Zheng, Olatunji Ruwase, Shaden Smith, Minjia Zhang, Jeff Rasley, et al. Deepspeed-inference: Enabling efficient inference of transformer models at unprecedented scale. In 2022 SC22: International Conference for High Performance Computing, Networking, Storage and Analysis (SC), pages 646-660. IEEE Computer Society, 2022.
- Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 1877-1901.
- Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. FlashAttention: Fast and memory-efficient exact attention with IO-awareness. In Advances in Neural Information Processing Systems, 2022.
- Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, and Jacob Steinhardt. Measuring coding challenge competence with apps. NeurIPS, 2021.
- Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Symposium on Operating Systems Principles, pages 611-626, 2023.
- Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, Thomas Hubert, Peter Choy, Cyprien de Masson d'Autume, Igor Babuschkin, Xinyun Chen, Po-Sen Huang, Johannes Welbl, Sven Gowal, Alexey Cherepanov, James Molloy, Daniel J. Mankowitz, Esme Sutherland Robson, Pushmeet Kohli, Nando de Freitas, Koray Kavukcuoglu, and Oriol Vinyals. Competition-level code generation with alphacode. Science, 378(6624):1092-1097, December 2022.
- Gyeong-In Yu, Joo Seong Jeong, Geon-Woo Kim, Soojeong Kim, and Byung-Gon Chun. Orca: A distributed serving system for {Transformer-Based} generative models. In 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22), pages 521-538, 2022.
Cited by
Sorted by citations
InstAttention: In-Storage Attention Offloading for Cost-Effective Long-Context LLM Inference
Pan, X., Li, E., Li, Q., Liang, S., Shan, Y., Zhou, K., Luo, Y., Wang, X., & Zhang, J. (2025) · IEEE International Symposium on High-Performance Computer Architecture (HPCA) · doi:10.1109/hpca61900.2025.00113 · Cited by 38 · see citing works
“As LLMs continue to evolve and push the boundaries toward longer context reasoning [26], [37] and larger batches [28], [57], the memory footprint of their associated KV cache in offline-inference escalates drastically [33], introducing substantial challenges in storing them efficiently.”
MoE-L ightning : High-Throughput MoE Inference on Memory-constrained GPUs
Cao, S., Liu, S., Griggs, T., Schafhalter, P., Liu, X., Sheng, Y., Gonzalez, J. E., Zaharia, M., & Stoica, I. G. (2025) · International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) · doi:10.1145/3669940.3707267 · Cited by 22 · see citing works
SGLang: Efficient Execution of Structured Language Model Programs
Zheng, L., Yin, L., Xie, Z., Sun, C., Huang, J., Yu, C. H., Cao, S., Kozyrakis, C., Stoica, I., Gonzalez, J. E., et al. (2023) · arXiv · doi:10.48550/arxiv.2312.07104 · Cited by 145 · see citing works
“HydraGen [18], FlashInfer [59], and ChunkedAttention [58] focus on CUDA kernel optimizations and do not include the concept of an LRU cache.”
Preble: Efficient Distributed Prompt Scheduling for LLM Serving
Srivatsa, V., He, Z., Abhyankar, R., Li, D., & Zhang, Y. (2024) · arXiv · doi:10.48550/arxiv.2407.00023 · Cited by 1 · see citing works
“To generate better-quality programs, an approach taken by a recent paper [18] is to add a demonstration of several generic code examples before the user problem to instruct an LLM.”
BackgroundKVDirect: Distributed Disaggregated LLM Inference
Chen, S., Jiang, R., Yu, D., Xu, J., Chao, M., Meng, F., Jiang, C., Xu, W., & Liu, H. (2025) · arXiv · doi:10.48550/arxiv.2501.14743 · Cited by 1 · see citing works
“Hydragen [17] proposes an implementation of attention to leverage the shared prefix within a batch of prompts.”
PEEK: Predictive Queue-Informed KV Cache Management for LLM Serving
Xie, B., Wang, Z., Tanaka, M., & Zheng, Z. (2026) · arXiv · doi:10.48550/arxiv.2607.02525
“…re-encoding), RAGCache [Jin et al., 2024] (hierarchical RAG cache), ChunkAttention [Ye et al., 2024] (attention over a prefix tree), Hydragen [Juravsky et al., 2024] (fused shared-prefix kernel), CachedAttention [Gao et al., 2024] (session-scoped state), CacheBlend [Yao et al., 2024] (block…”
Don't Break the Cache: An Evaluation of Prompt Caching for Long-Horizon Agentic Tasks
Lumer, E., Nizar, F., Jangiti, A., Frank, K., Gulati, A., Phadate, M., & Subbiah, V. K. (2026) · arXiv · doi:10.48550/arxiv.2601.06007
“…selec-2 tive retention of important tokens (Ge et al., 2023), storage-compute tradeoffs that balance recomputation against cache loading (Jin et al., 2024), and shared prefix optimization for high-throughput inference (Juravsky et al., 2024; Wu et al., 2024a; Sun et al., 2025; Zhou et al., 2024).”
- Authors
- Juravsky, J., Brown, B., Ehrlich, R., Fu, D. Y., Ré, C., & Mirhoseini, A.
- Affiliations
- Stanford University (US)
- University of Oxford (GB)
- University of Waterloo (CA)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2024
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2402.05099
- OpenAlex
- W4391670445
- Cited by
- 2
- References
- 0
- Citations by year
- 2025: 2
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Funders
- National Science Foundation (1763315, 1563078, 2247015, 1937301)
- Accenture
- VMware
- National Institutes of Health (N000141712266)
- Office of Naval Research (N000141712266)
- National Institute of Biomedical Imaging and Bioengineering (U54EB020405)
- U.S. Army Combat Capabilities Development Command
- Topics
- Physical Sciences › Computer Science › Hardware and Architecture › Parallel Computing and Optimization Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- large language model inference, key-value cache, high-throughput inference, matrix multiplication, CodeLlama, memory efficiency
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
