Efficient Memory Management for Large Language Model Serving with PagedAttention
Open accessThe DOI matches its Crossref record; 46 of the 46 references it cites were found in a registry or an index; 20 of the 35 sources its claims cite were checked for support.
Abstract
High throughput serving of large language models (LLMs) requires batching sufficiently many requests at a time. However, existing systems struggle because the key-value cache (KV cache) memory for each request is huge and grows and shrinks dynamically. When managed inefficiently, this memory can be significantly wasted by fragmentation and redundant duplication, limiting the batch size. To address this problem, we propose PagedAttention, an attention algorithm inspired by the classical virtual memory and paging techniques in operating systems. On top of it, we build vLLM, an LLM serving system that achieves (1) near-zero waste in KV cache memory and (2) flexible sharing of KV cache within and across requests to further reduce memory usage. Our evaluations show that vLLM improves the throughput of popular LLMs by 2--4× with the same level of latency compared to the state-of-the-art systems, such as FasterTransformer and Orca. The improvement is more pronounced with longer sequences, larger models, and more complex decoding algorithms. vLLM's source code is publicly available at https://github.com/vllm-project/vllm.
Overview
CoChat Summary
TL;DR
The paper introduces PagedAttention, an attention algorithm that stores KV cache in non-contiguous fixed-size blocks inspired by OS virtual memory and paging, and builds vLLM, an LLM serving system with paged KV cache management and sharing. Evaluations show vLLM improves LLM serving throughput by 2–4× over FasterTransformer and Orca at similar latency, with larger gains for longer sequences, larger models, and complex decoding.
Key findings
- vLLM improves throughput by 2–4× over state-of-the-art systems FasterTransformer and Orca with the same latency; on ShareGPT it sustains 1.7–2.7× higher request rates than Orca (Oracle) and 2.7–8× than Orca (Max), and up to 22× vs FasterTransformer.
- PagedAttention enables memory sharing: on Alpaca, 6.1–9.8% memory saving for parallel sampling and 37.6–55.2% for beam search; on ShareGPT, 16.2–30.5% and 44.3–66.3% respectively. Shared prefix gives 1.67× higher throughput than Orca (Oracle) with a one-shot prefix and 3.58× with a few-shot prefix; chatbot workload sustains 2× higher request rates.
- Existing systems waste KV cache memory: only 20.4–38.2% of KV cache memory is used to store actual token states, while vLLM achieves near-zero waste. PagedAttention adds 20–26% higher attention kernel latency vs FasterTransformer, but vLLM still outperforms end-to-end.
- vLLM uses default block size 16; recomputation overhead is never higher than 20% of swapping’s latency, and for medium block sizes 16–64 the two recovery methods show comparable end-to-end performance.
Methods
- Design: PagedAttention partitions each sequence’s KV cache into fixed-size KV blocks, stores them in non-contiguous GPU memory, and uses block tables mapping logical to physical blocks; vLLM adds a centralized scheduler, KV cache manager, copy-on-write, preemption via recomputation or swapping, and distributed execution with Megatron-LM style tensor parallelism.
- Evaluation: OPT 13B/66B/175B and LLaMA 13B on NVIDIA A100 GPUs (A2 instances on Google Cloud Platform); synthetic workloads from ShareGPT and Alpaca with Poisson request arrivals; baselines are FasterTransformer with a custom dynamic-batching scheduler and three Orca variants (Oracle, Pow2, Max); metrics are normalized latency and throughput.
- Ablations: kernel microbenchmark, block size impact, and recomputation vs swapping; most experiments use 1-hour traces, except OPT-175B uses 15-minute traces due to cost.
Limitations
- Stated: virtual memory and paging are effective for LLM serving because the workload requires dynamic memory allocation and is memory-bound, but this does not generally hold for every GPU workload; DNN training has static shapes, and non-LLM serving may be compute-bound, where vLLM’s techniques may degrade performance due to memory indirection and non-contiguous block memory.
- Stated/evident: Orca is not publicly available, so the authors implement their own Orca baselines; Orca (Oracle) assumes knowledge of actual output lengths and is infeasible in practice. Evident: evaluation uses synthetic request arrivals (Poisson) from ShareGPT and Alpaca without timestamps, and OPT-175B uses 15-minute traces due to cost.
- Evident: the chatbot evaluation does not store KV cache between different conversation rounds, which may limit applicability to multi-turn caching scenarios.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
3 Valid · 16 Not valid · 6 Not checked · 111 Own results and interpretations
The cited sources that could be checked couldn’t confirm or dispute this claim.
In the paper
“for the 13B parameter OPT model [62], the KV cache of a single token demands 800 KB of space, calculated as 2 (key and value vectors) × 5120 (hidden state size) × 40 (number of layers) × 2 (bytes per FP16).”
§ Efficient Memory Management for Large Language Model Serving with PagedAttention > 3 Memory Challenges in LLM Serving
What the sources say
- Could not verifycites [62] matched to “Alpa: Automating Inter- and Intra-Operator Parallelism for Distributed Deep Learning” (2022) sourceExperimental — accuracy not yet measured
The model specifications (Appendix B, p. 19) list GPT-3 models, not OPT-13B; Table 6 (p. 19) gives a 5120-wide GPT-3 model with 48 layers, not the claimed OPT configuration. Searches for OPT-13B and per-token key–value cache memory found no passage addressing the 800 KB figure or its calculation. (LLM, conf=0.76)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Could not verify
The cited sources that could be checked couldn’t confirm or dispute this claim.
In the paper
“First, the existing systems [31,60] suffer from internal and external memory fragmentation.”
§ Efficient Memory Management for Large Language Model Serving with PagedAttention > 1 Introduction
What the sources say
- Could not verifycites [31] 31. Christopher Olston, Noah Fiedel, Kiril Gorovoy, Jeremiah Harmsen, Li Lao, Fangwei Li, Vinu Rajashekhar, Sukriti Ramesh, and Jordan Soyke. 2017. Tensorflow-serving: Flexible, high-performance ml serving. arXiv preprint arXiv:1712.06139 (2017). sourceExperimental — accuracy not yet measured
The Introduction (p. 1) discusses managing server RAM as models grow, and §2.1.2 (p. 4) describes unloading models to free memory. Neither passage addresses internal or external memory fragmentation, and searches for those terms found no passage that does. The copy also does not discuss LLM serving systems. (LLM, conf=0.72)
- Not matchedcites [60]
The reference list we hold has no entry [60].
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Could not verify
The cited sources that could be checked couldn’t confirm or dispute this claim.
In the paper
“In more sophisticated algorithms like beam search [49], different request beams can share larger portions (up to 55% memory saving, see §6.3) of their KV cache”
§ Efficient Memory Management for Large Language Model Serving with PagedAttention > 3 Memory Challenges in LLM Serving
What the sources say
- Could not verifycites [49] 49. Jing Wang, Youyou Lu, Qing Wang, Minhui Xie, Keji Huang, and Jiwu Shu. 2022. Pacman: An Efficient Compaction Approach for {Log- Structured} {Key-Value} Store on Persistent Memory. In 2022 USENIX Annual Technical Conference (USENIX ATC 22). 773-788. sourceExperimental — accuracy not yet measured
The passages from our copy discuss persistent-memory key-value stores, not beam search or language-model KV caches. Searches for beam sharing, KV-cache sharing, and the claimed 55% memory saving found no passage addressing the claim; §6.3 (p. 12) instead reports storage-system performance. (LLM, conf=0.76)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Could not verify
The cited sources that could be checked couldn’t confirm or dispute this claim.
In the paper
“Due to the unpredictable output lengths from the LLM, they statically allocate a chunk of memory for a request based on the request's maximum possible sequence length, irrespective of the actual input or eventual output length of the request.”
§ Efficient Memory Management for Large Language Model Serving with PagedAttention > 3.1 Memory Management in Existing Systems
What the sources say
- Could not verifycites [31] 31. Christopher Olston, Noah Fiedel, Kiril Gorovoy, Jeremiah Harmsen, Li Lao, Fangwei Li, Vinu Rajashekhar, Sukriti Ramesh, and Jordan Soyke. 2017. Tensorflow-serving: Flexible, high-performance ml serving. arXiv preprint arXiv:1712.06139 (2017). sourceExperimental — accuracy not yet measured
Section 2.2.1 (p. 5) discusses batching inference requests, while the Introduction (p. 1) discusses managing memory for loaded models. Neither addresses static memory allocation per request based on maximum sequence length or unpredictable LLM output length. Searches for those terms and their paraphrases found no supporting passage in our copy. (LLM, conf=0.75)
- Not matchedcites [60]
The reference list we hold has no entry [60].
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Could not verify
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“However, these general systems fail to take into account the autoregressive property and token state of LLM inference, resulting in missed opportunities for optimization.”
§ Efficient Memory Management for Large Language Model Serving with PagedAttention > 8 Discussion
What the sources say
- Partly supportedcites [11] 11. Daniel Crankshaw, Xin Wang, Guilio Zhou, Michael J Franklin, Joseph E Gonzalez, and Ion Stoica. 2017. Clipper: A Low-Latency Online Prediction Serving System. In 14th USENIX Symposium on Networked Systems Design and Implementation (NSDI 17). 613-627. sourceExperimental — accuracy not yet measured
Section 7 (Limitations; page not available) says Clipper treats models as black boxes and does not optimize their execution, supporting the general point that its serving architecture misses model-specific optimizations. The paper does not address LLM inference, autoregressive generation, or token state, and it does not establish the claim about general serving systems as a group. (LLM, conf=0.78)
- Could not verifycites [33] 33. OpenAI. 2022. https://openai.com/blog/chatgpt sourceExperimental — accuracy not yet measured
The Inference section (p. 4) discusses planned APIs for sequence models, and Inter-Request Batching (p. 5) describes batching within a sequence model’s loop. Neither passage addresses LLM token state or concludes that general serving systems miss optimizations because they ignore it (searched: autoregressive token state, sequence-model decoding and batching). (LLM, conf=0.76)
- Partly supportedcites [45] 45. Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014. Sequence to se- quence learning with neural networks. Advances in neural information processing systems 27 (2014). sourceExperimental — accuracy not yet measured
Related Work says that inference systems inheriting offloading techniques from training ignore the special computational properties of generative inference and miss opportunities to schedule I/O efficiently. The Generative Inference section describes token-by-token dependencies and the KV cache. The paper makes this criticism of offloading-based inference systems, not general model-serving systems as a whole, and does not specifically say those systems ignore token state. (LLM, conf=0.82)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
72 more claims
Create a free account to see all 136 claims and to verify the 1 reference still open.
- Probably not a substantial claim. 13
- It may state two separate claims as one. 48
- The quote only partly states this claim. 2
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source doesn’t support the claim as written.
In the citing paper
“vLLM [23] and ChunkedAttention [58] explore some simple reuse cases (e.g., system prompt sharing) but do not cover multi-level tree-structured sharing or LRU caching.”
From SGLang: Efficient Execution of Structured Language Model Programs (2023) · cites this paper as [23]
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“vLLM [30] reserves a fixed set of physical memory blocks for shared prefixes, enabling reuse of prefixes that align with predefined block sizes.”
From HotPrefix: Hotness-Aware KV Cache Scheduling for Efficient Prefix Sharing in LLM Inference Systems (2025) · cites this paper as [30]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Existing methods [30,72] often fail to effectively manage GPU memory, resulting in frequent eviction of high-hotness caches in favor of low-hotness ones.”
From HotPrefix: Hotness-Aware KV Cache Scheduling for Efficient Prefix Sharing in LLM Inference Systems (2025) · cites this paper as [30]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“While this mechanism simplifies memory allocation, it suffers from coarse granularity.If the prefix length does not align with block sizes, some parts of the prefix cannot be effectively reused, leading to inefficiencies.”
From HotPrefix: Hotness-Aware KV Cache Scheduling for Efficient Prefix Sharing in LLM Inference Systems (2025) · cites this paper as [30]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Parallel efforts target memory efficiency, such as PagedAttention's approach to KV cache management, which reduces fragmentation and overhead [289].”
From A Survey of Large Language Models (2026) · cites this paper as [289]
Experimental — accuracy not yet measured
72 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · cited 1×
“4, all other components in the Transformer model, including the embedding layer, feed-forward layer, layer normalization [2], residual connection [22], output logit computation, and the query, key, and value transformation in Eq.”
BackgroundVaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
PyTorch: An Imperative Style, High-Performance Deep Learning Library
Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., et al. (2019) · arXiv · doi:10.48550/arxiv.1912.01703 · Cited by 31,473
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · cited 19×
“The emergence of large language models ( LLMs ) like GPT [5, 37] and PaLM [9] have enabled new applications such as programming assistants [6, 18] and universal chatbots [19, 35] that are starting to profoundly impact our work and daily routines.”
InfluentialTransformers: State-of-the-Art Natural Language Processing
Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., et al. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations · doi:10.18653/v1/2020.emnlp-demos.6 · Cited by 8,741
Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
Wu, Y., Schuster, M., Chen, Z., Le, Q. V., Norouzi, M., Macherey, W., Krikun, M., Cao, Y., Gao, Q., Macherey, K., et al. (2016) · arXiv · doi:10.48550/arxiv.1609.08144 · Cited by 5,319 · cited 1×
“In LLM tasks like machine translation [59], the users expect the top-𝑘 most appropriate translations output by the LLM. Beam search [49] is widely used to decode the most probable output sequence from an LLM, as it mitigates the computational complexity of fully traversing the Unlike parallel…”
LLaMA: Open and Efficient Foundation Language Models
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. (2023) · arXiv · doi:10.48550/arxiv.2302.13971 · Cited by 4,602
HuggingFace's Transformers: State-of-the-art Natural Language Processing
Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., et al. (2019) · arXiv · doi:10.48550/arxiv.1910.03771 · Cited by 3,347
The Power of Scale for Parameter-Efficient Prompt Tuning
Lester, B., Al-Rfou, R., & Constant, N. (2021) · arXiv · doi:10.48550/arxiv.2104.08691 · Cited by 3,092
2017 USENIX Annual Technical Conference (USENIX ATC'17)
Quoc, D. L., Beck, M. S., Bhatotia, P., Chen, R., Fetzer, C. W., & Strufe, T. (2017) · Cited by 3,091
PaLM: Scaling Language Modeling with Pathways
Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2204.02311 · Cited by 2,590
Prefix-Tuning: Optimizing Continuous Prompts for Generation
Li, X. L., & Liang, P. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.353 · Cited by 2,473
Evaluating Large Language Models Trained on Code
Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. D. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. (2021) · arXiv · doi:10.48550/arxiv.2107.03374 · Cited by 1,436
Self-Instruct: Aligning Language Models with Self-Generated Instructions
Wang, Y., Kordi, Y., Mishra, S., Liu, A., Smith, N. A., Khashabi, D., & Hajishirzi, H. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.acl-long.754 · Cited by 1,243
FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness
Dao, T., Fu, D. Y., Ermon, S., Rudra, A., & Ré, C. (2022) · arXiv · doi:10.48550/arxiv.2205.14135 · Cited by 1,214
Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism
Shoeybi, M., Patwary, M., Puri, R., LeGresley, P., Casper, J., & Catanzaro, B. (2019) · arXiv · doi:10.48550/arxiv.1909.08053 · Cited by 893
Training Deep Nets with Sublinear Memory Cost
Chen, T., Xu, B., Zhang, C., & Guestrin, C. (2016) · arXiv · doi:10.48550/arxiv.1604.06174 · Cited by 561
Findings of the 2016 Conference on Machine Translation
Bojar, O., Chatterjee, R., Federmann, C., Graham, Y., Haddow, B., Huck, M., Jimeno Yepes, A., Koehn, P., Logacheva, V., Monz, C., et al. (2016) · Proceedings of the First Conference on Machine Translation: Volume 2, Shared Task Papers · doi:10.18653/v1/w16-2301 · Cited by 539
Proceedings of the 10th USENIX conference on Operating Systems Design and Implementation
Vahdat, A. M., & Thekkath, C. (2012) · Cited by 454
Ba, J. L., Kiros, J., & Hinton, G. E. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1607.06450 · Cited by 440
Clipper: A Low-Latency Online Prediction Serving System
Crankshaw, D., Wang, X., Zhou, G., Franklin, M. J., Gonzalez, J. E., & Stoica, I. (2016) · arXiv · doi:10.48550/arxiv.1612.03079 · Cited by 415
Self-citationGholami, A., Yao, Z., Kim, S., Hooper, C., Mahoney, M. W., & Keutzer, K. (2024) · IEEE Micro · doi:10.1109/mm.2024.3373763 · Cited by 369
AlpaServe: Statistical Multiplexing with Model Parallelism for Deep Learning ServingPreprint
Li, Z., Zheng, L., Zhong, Y., Liu, V., Sheng, Y., Jin, X., Huang, Y., Chen, Z., Zhang, H., Gonzalez, J. E., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2302.11665 · Cited by 322
Self-citationDeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scale
Aminabadi, R. Y., Rajbhandari, S., Zhang, M., Awan, A. A., Li, C., Li, D., Zheng, E., Rasley, J., Smith, S., Ruwase, O., et al. (2022) · arXiv · doi:10.48550/arxiv.2207.00032 · Cited by 297
Prefix-Tuning: Optimizing Continuous Prompts for Generation
Li, X. L., & Liang, P. (2021) · arXiv · doi:10.48550/arxiv.2101.00190 · Cited by 289
Kilburn, T., Edwards, D. B. G., Lanigan, M. J., & Sumner, F. H. (1962) · IEEE Transactions on Electronic Computers · doi:10.1109/tec.1962.5219356 · Cited by 256 · cited 2×
“To address the memory challenges in §3, we introduce PagedAttention, an attention algorithm inspired by the classic idea of paging [25] in operating systems.”
BackgroundMethodNexus: a GPU cluster engine for accelerating DNN-based video analysis
Shen, H., Chen, L., Jin, Y., Zhao, L., Kong, B., Philipose, M., Krishnamurthy, A., & Sundaram, R. (2019) · Proceedings of the 27th ACM Symposium on Operating Systems Principles · doi:10.1145/3341301.3359658 · Cited by 232 · cited 1×
“Clipper [11], TensorFlow Serving [33], Nexus [45], InferLine [10], and Clockwork [20] are some earlier general model serving systems.”
BackgroundSuperneurons: dynamic GPU memory management for training deep neural networks
Wang, L., Ye, J., Zhao, Y., Wu, W., Li, A., Song, S. L., Xu, Z., & Kraska, T. (2018) · Proceedings of the 23rd ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming · doi:10.1145/3178487.3178491 · Cited by 175 · cited 1×
“Swapping [23, 42, 55], recomputation [7, 24] and their combination [40] have been utilized to reduce the peak memory of training.”
MethodTensorFlow-Serving: Flexible, High-Performance ML Serving
Olston, C., Fiedel, N., Gorovoy, K., Harmsen, J., Lao, L., Li, F., Rajashekhar, V., Ramesh, S., & Soyke, J. (2017) · arXiv · doi:10.48550/arxiv.1712.06139 · Cited by 171
SwapAdvisor: Pushing Deep Learning Beyond the GPU Memory Limit via Smart Swapping
Huang, C.-C., Jin, G., & Li, J. (2020) · Proceedings of the Twenty-Fifth International Conference on Architectural Support for Programming Languages and Operating Systems · doi:10.1145/3373376.3378530 · Cited by 158 · cited 1×
“Swapping [23, 42, 55], recomputation [7, 24] and their combination [40] have been utilized to reduce the peak memory of training.”
MethodZeRO-Offload: Democratizing Billion-Scale Model Training
Ren, J., Rajbhandari, S., Aminabadi, R. Y., Ruwase, O., Yang, S., Zhang, M., Li, D., & He, Y. (2021) · arXiv · doi:10.48550/arxiv.2101.06840 · Cited by 151 · cited 1×
“Swapping [23, 42, 55], recomputation [7, 24] and their combination [40] have been utilized to reduce the peak memory of training.”
MethodAlpa: Automating Inter- and Intra-Operator Parallelism for Distributed Deep Learning
Zheng, L., Li, Z., Zhang, H., Zhuang, Y., Chen, Z., Huang, Y., Wang, Y., Xu, Y., Zhuo, D., Xing, E. P., et al. (2022) · arXiv · doi:10.48550/arxiv.2201.12023 · Cited by 124
Self-citationSelf-Instruct: Aligning Language Models with Self-Generated Instructions
Wang, Y., Kordi, Y., Mishra, S., Liu, A., Smith, N. A., Khashabi, D., & Hajishirzi, H. (2022) · arXiv · doi:10.48550/arxiv.2212.10560 · Cited by 118
TurboTransformers: an efficient GPU serving system for transformer models
Fang, J., Yu, Y., Zhao, C., & Zhou, J. (2021) · Proceedings of the 26th ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming · doi:10.1145/3437801.3441578 · Cited by 116 · cited 1×
“These systems utilize GPU kernel optimizations [1, 29, 31, 56], advanced batching mechanisms [14, 60], model parallelism [1, 41, 60], and parameter sharing [64] for efficient serving.”
MethodEfficiently Scaling Transformer Inference
Pope, R., Douglas, S., Chowdhery, A., Devlin, J., Bradbury, J., Levskaya, A., Heek, J., Xiao, K., Agrawal, S., & Dean, J. (2022) · arXiv · doi:10.48550/arxiv.2211.05102 · Cited by 113
InferLine: latency-aware provisioning and scaling for prediction serving pipelines
Crankshaw, D., Sela, G.-E., Mo, X., Zumar, C., Stoica, I., Gonzalez, J., & Tumanov, A. (2020) · Proceedings of the 11th ACM Symposium on Cloud Computing · doi:10.1145/3419111.3421285 · Cited by 104 · cited 1×
“Clipper [11], TensorFlow Serving [33], Nexus [45], InferLine [10], and Clockwork [20] are some earlier general model serving systems.”
BackgroundSelf-citationFlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPU
Sheng, Y., Zheng, L., Yuan, B., Li, Z., Ryabinin, M., Fu, D. Y., Xie, Z., Chen, B., Barrett, C., Gonzalez, J. E., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.06865 · Cited by 86 · cited 2×
“Notably, FlexGen [46] studies how to swap weights and token states for LLM inference with limited GPU memory, but it does not target the online serving settings.”
BackgroundSelf-citationServing DNNs like Clockwork: Performance Predictability from the Bottom Up
Gujarati, A., Karimi, R. J., Alzayat, S., Hao, W., Kaufmann, A., Vigfússon, Ý., & Mace, J. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.02464 · Cited by 70
Low latency RNN inference with cellular batching
Gao, P., Yu, L., Wu, Y., & Li, J. (2018) · Proceedings of the Thirteenth EuroSys Conference · doi:10.1145/3190508.3190541 · Cited by 66 · cited 1×
“To address this problem, fine-grained batching mechanisms, such as cellular batching [16] and iteration-level scheduling [60], have been proposed.”
Superneurons: dynamic GPU memory management for training deep neural networks
Wang, L., Ye, J., Zhao, Y., Wu, W., Li, A., Song, S. L., Xu, Z., & Kraska, T. (2018) · ACM SIGPLAN Notices · doi:10.1145/3200691.3178491 · Cited by 62
POET: Training Neural Networks on Tiny Devices with Integrated Rematerialization and PagingPreprint
Patil, S. G., Jain, P., Dutta, P., Stoica, I. G., & Gonzalez, J. E. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2207.07697 · Cited by 53
Self-citationLightSeq: A High Performance Inference Library for Transformers
Wang, X., Xiong, Y., Wei, Y., Wang, M., & Li, L. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies: Industry Papers · doi:10.18653/v1/2021.naacl-industry.15 · Cited by 36 · cited 1×
“These systems utilize GPU kernel optimizations [1, 29, 31, 56], advanced batching mechanisms [14, 60], model parallelism [1, 41, 60], and parameter sharing [64] for efficient serving.”
Checkmate: Breaking the Memory Wall with Optimal Tensor Rematerialization
Jain, P., Jain, A., Nrusimha, A., Gholami, A., Abbeel, P., Keutzer, K., Stoica, I. G., & Gonzalez, J. E. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1910.02653 · Cited by 34 · cited 1×
“Swapping [23, 42, 55], recomputation [7, 24] and their combination [40] have been utilized to reduce the peak memory of training.”
MethodSelf-citationOLLA: Optimizing the Lifetime and Location of Arrays to Reduce the Memory Usage of Neural NetworksPreprint
Steiner, B., Elhoushi, M., Kahn, J., & Hegarty, J. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2210.12924 · Cited by 17 · cited 2×
“OLLA [48] optimizes the lifetime and location of tensors to reduce fragmentation, but it does not do finegrained block-level management or online serving.”
BackgroundOpenAI, Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.08774
OPT: Open Pre-trained Transformer Language Models
Zhang, S., Roller, S., Goyal, N., Artetxe, M., Chen, M., Chen, S., Dewan, C., Diab, M., Li, X., Lin, X. V., et al. (2022) · arXiv · doi:10.48550/arxiv.2205.01068
Research works printed but not linked: 8
- Yoshua Bengio , Réjean Ducharme , and Pascal Vincent . 2000. A neural probabilistic language model. Advances in neural information processing systems 13 ( 2000 ). Yoshua Bengio, Réjean Ducharme, and Pascal Vincent. 2000. A neural probabilistic language model. Advances in neural information processing systems 13 (2000).
- Weihao Cui , Han Zhao , Quan Chen , Hao Wei , Zirui Li , Deze Zeng , Chao Li , and Minyi Guo . 2022 . DVABatch: Diversity-aware Multi-Entry Multi-Exit Batching for Efficient Processing of DNN Services on GPUs. In 2022 USENIX Annual Technical Conference (USENIX ATC 22) . 183--198. Weihao Cui, Han Zhao, Quan Chen, Hao Wei, Zirui Li, Deze Zeng, Chao Li, and Minyi Guo. 2022. DVABatch: Diversity-aware Multi-Entry Multi-Exit Batching for Efficient Processing of DNN Services on GPUs. In 2022 USENIX Annual Technical Conference (USENIX ATC 22). 183--198.
- Mingcong Han , Hanze Zhang , Rong Chen , and Haibo Chen . 2022 . Microsecond-scale Preemption for Concurrent {GPU-accelerated}{DNN} Inferences . In 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22) . 539--558. Mingcong Han, Hanze Zhang, Rong Chen, and Haibo Chen. 2022. Microsecond-scale Preemption for Concurrent {GPU-accelerated}{DNN} Inferences. In 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22). 539--558.
- Ilya Sutskever , Oriol Vinyals , and Quoc V Le. 2014. Sequence to sequence learning with neural networks. Advances in neural information processing systems 27 ( 2014 ). Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014. Sequence to sequence learning with neural networks. Advances in neural information processing systems 27 (2014).
- Jing Wang , Youyou Lu , Qing Wang , Minhui Xie , Keji Huang , and Jiwu Shu . 2022 . Pacman: An Efficient Compaction Approach for {Log-Structured} {Key-Value} Store on Persistent Memory . In 2022 USENIX Annual Technical Conference (USENIX ATC 22) . 773--788. Jing Wang, Youyou Lu, Qing Wang, Minhui Xie, Keji Huang, and Jiwu Shu. 2022. Pacman: An Efficient Compaction Approach for {Log-Structured} {Key-Value} Store on Persistent Memory. In 2022 USENIX Annual Technical Conference (USENIX ATC 22). 773--788.
- Gyeong-In Yu , Joo Seong Jeong , Geon-Woo Kim , Soojeong Kim , and Byung-Gon Chun . 2022 . Orca: A Distributed Serving System for {Transformer-Based} Generative Models . In 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22) . 521--538. Gyeong-In Yu, Joo Seong Jeong, Geon-Woo Kim, Soojeong Kim, and Byung-Gon Chun. 2022. Orca: A Distributed Serving System for {Transformer-Based} Generative Models. In 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22). 521--538.
- Hong Zhang , Yupeng Tang , Anurag Khandelwal , and Ion Stoica . 2023 . SHEPHERD: Serving DNNs in the Wild. In 20th USENIX Symposium on Networked Systems Design and Implementation (NSDI 23) . USENIX Association, Boston, MA, 787--808. https://www.usenix.org/conference/nsdi23/presentation/zhang-hong Hong Zhang, Yupeng Tang, Anurag Khandelwal, and Ion Stoica. 2023. SHEPHERD: Serving DNNs in the Wild. In 20th USENIX Symposium on Networked Systems Design and Implementation (NSDI 23). USENIX Association, Boston, MA, 787--808. https://www.usenix.org/conference/nsdi23/presentation/zhang-hong
- Zhe Zhou , Xuechao Wei , Jiejing Zhang , and Guangyu Sun . 2022 . PetS: A Unified Framework for Parameter-Efficient Transformers Serving. In 2022 USENIX Annual Technical Conference (USENIX ATC 22) . 489--504. Zhe Zhou, Xuechao Wei, Jiejing Zhang, and Guangyu Sun. 2022. PetS: A Unified Framework for Parameter-Efficient Transformers Serving. In 2022 USENIX Annual Technical Conference (USENIX ATC 22). 489--504.
Cited by
A Survey of Large Language Models
Zhao, W. X., Zhou, K., Li, J., Tang, T., Dong, Z., Hou, Y., Zhang, B., Min, Y., Zhang, J., Liu, P., et al. (2026) · Frontiers of Computer Science · doi:10.1007/s11704-026-60308-3 · Cited by 2,093
A Comprehensive Overview of Large Language Models
Naveed, H., Khan, A. U., Qiu, S., Saqib, M., Anwar, S., Usman, M., Akhtar, N., Barnes, N., & Mian, A. (2023) · arXiv · doi:10.48550/arxiv.2307.06435 · Cited by 1,259
A Comprehensive Overview of Large Language Models
Naveed, H., Khan, A. U., Qiu, S., Saqib, M., Anwar, S., Usman, M., Akhtar, N., Barnes, N., & Mian, A. (2025) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3744746 · Cited by 1,257
DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
DeepSeek-AI, Guo, D., Yang, D., Zhang, H., Song, J., Wang, P., Zhu, Q., Xu, R., Zhang, R., Ma, S., et al. (2025) · arXiv · doi:10.48550/arxiv.2501.12948 · Cited by 18,918
Self-citationJiang, A. Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D. S., Casas, D. D. L., Bressand, F., Lengyel, G., Lample, G., Saulnier, L., et al. (2023) · arXiv · doi:10.48550/arxiv.2310.06825 · Cited by 340
“This release is accompanied by a reference implementation1 facilitating easy deployment either locally or on cloud platforms such as AWS, GCP, or Azure using the vLLM [17] inference server and SkyPilot 2.”
Method
Metadata
- Authors
- Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez, J., Zhang, H., & Stoica, I.
- Affiliations
- University of California, Berkeley (US)
- Independent Researcher, Berkeley, United States of America
- University of California San Diego (US)
- ORCID
- Venue
- Proceedings of the 29th Symposium on Operating Systems Principles
- Publisher
- ACM
- Published
- 2023-10-23
- Pages
- 611-626
- OpenAlex
- W4387321091
- Semantic Scholar
- 83b90f4a0ae4cc214eb3cc140ccfef9cd99fac05
- Cited by
- 1,817
- References
- 19
- Citations by year
- 2026: 1017 · 2025: 623 · 2024: 169 · 2023: 8
- FWCI
- 321.20
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- ACM SIGOPS Symposium on Operating Systems Principles (SOSP) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Funders
- VMware
- Zayed University
- Samsung
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Hardware and Architecture › Parallel Computing and Optimization Techniques
- Keywords
- large language model serving, memory management, key-value cache, request batching, memory fragmentation, vLLM, inference throughput, dynamic memory allocation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-09-28, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-09-28 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.