Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its Crossref record; 46 of the 46 references it cites were found in a registry or an index; 20 of the 35 sources its claims cite were checked for support.

Efficient Memory Management for Large Language Model Serving with PagedAttention

Open access
Kwon, Woosuk, Li, Zhuohan, Zhuang, Siyuan, Sheng, Ying, Zheng, Lianmin, Yu, Cody Hao, Gonzalez, Joseph, Zhang, Hao, Stoica, Ion · 2023 · Proceedings of the 29th Symposium on Operating Systems Principles

Abstract

High throughput serving of large language models (LLMs) requires batching sufficiently many requests at a time. However, existing systems struggle because the key-value cache (KV cache) memory for each request is huge and grows and shrinks dynamically. When managed inefficiently, this memory can be significantly wasted by fragmentation and redundant duplication, limiting the batch size. To address this problem, we propose PagedAttention, an attention algorithm inspired by the classical virtual memory and paging techniques in operating systems. On top of it, we build vLLM, an LLM serving system that achieves (1) near-zero waste in KV cache memory and (2) flexible sharing of KV cache within and across requests to further reduce memory usage. Our evaluations show that vLLM improves the throughput of popular LLMs by 2--4× with the same level of latency compared to the state-of-the-art systems, such as FasterTransformer and Orca. The improvement is more pronounced with longer sequences, larger models, and more complex decoding algorithms. vLLM's source code is publicly available at https://github.com/vllm-project/vllm.

Overview

CoChat Summary

TL;DR

The paper introduces PagedAttention, an attention algorithm that stores KV cache in non-contiguous fixed-size blocks inspired by OS virtual memory and paging, and builds vLLM, an LLM serving system with paged KV cache management and sharing. Evaluations show vLLM improves LLM serving throughput by 2–4× over FasterTransformer and Orca at similar latency, with larger gains for longer sequences, larger models, and complex decoding.

Key findings

  • vLLM improves throughput by 2–4× over state-of-the-art systems FasterTransformer and Orca with the same latency; on ShareGPT it sustains 1.7–2.7× higher request rates than Orca (Oracle) and 2.7–8× than Orca (Max), and up to 22× vs FasterTransformer.
  • PagedAttention enables memory sharing: on Alpaca, 6.1–9.8% memory saving for parallel sampling and 37.6–55.2% for beam search; on ShareGPT, 16.2–30.5% and 44.3–66.3% respectively. Shared prefix gives 1.67× higher throughput than Orca (Oracle) with a one-shot prefix and 3.58× with a few-shot prefix; chatbot workload sustains 2× higher request rates.
  • Existing systems waste KV cache memory: only 20.4–38.2% of KV cache memory is used to store actual token states, while vLLM achieves near-zero waste. PagedAttention adds 20–26% higher attention kernel latency vs FasterTransformer, but vLLM still outperforms end-to-end.
  • vLLM uses default block size 16; recomputation overhead is never higher than 20% of swapping’s latency, and for medium block sizes 16–64 the two recovery methods show comparable end-to-end performance.

Methods

  • Design: PagedAttention partitions each sequence’s KV cache into fixed-size KV blocks, stores them in non-contiguous GPU memory, and uses block tables mapping logical to physical blocks; vLLM adds a centralized scheduler, KV cache manager, copy-on-write, preemption via recomputation or swapping, and distributed execution with Megatron-LM style tensor parallelism.
  • Evaluation: OPT 13B/66B/175B and LLaMA 13B on NVIDIA A100 GPUs (A2 instances on Google Cloud Platform); synthetic workloads from ShareGPT and Alpaca with Poisson request arrivals; baselines are FasterTransformer with a custom dynamic-batching scheduler and three Orca variants (Oracle, Pow2, Max); metrics are normalized latency and throughput.
  • Ablations: kernel microbenchmark, block size impact, and recomputation vs swapping; most experiments use 1-hour traces, except OPT-175B uses 15-minute traces due to cost.

Limitations

  • Stated: virtual memory and paging are effective for LLM serving because the workload requires dynamic memory allocation and is memory-bound, but this does not generally hold for every GPU workload; DNN training has static shapes, and non-LLM serving may be compute-bound, where vLLM’s techniques may degrade performance due to memory indirection and non-contiguous block memory.
  • Stated/evident: Orca is not publicly available, so the authors implement their own Orca baselines; Orca (Oracle) assumes knowledge of actual output lengths and is infeasible in practice. Evident: evaluation uses synthetic request arrivals (Poisson) from ShareGPT and Alpaca without timestamps, and OPT-175B uses 15-minute traces due to cost.
  • Evident: the chatbot evaluation does not store KV cache between different conversation rounds, which may limit applicability to multi-turn caching scenarios.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1145/3600006.3613165

Cites

46 of 46 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Verification

Claims the paper makes

3 Valid · 16 Not valid · 6 Not checked · 111 Own results and interpretations

72 more claims

Create a free account to see all 136 claims and to verify the 1 reference still open.

See all claims — free account
  • Probably not a substantial claim. 13
  • It may state two separate claims as one. 48
  • The quote only partly states this claim. 2

Create a free account to see these claims.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

72 more claims about this paper

Create a free account to see every claim other papers make about this one.

See all claims — free account

Citations

Cites

Show all 19 references

Research works printed but not linked: 8

  1. Yoshua Bengio , Réjean Ducharme , and Pascal Vincent . 2000. A neural probabilistic language model. Advances in neural information processing systems 13 ( 2000 ). Yoshua Bengio, Réjean Ducharme, and Pascal Vincent. 2000. A neural probabilistic language model. Advances in neural information processing systems 13 (2000).
  2. Weihao Cui , Han Zhao , Quan Chen , Hao Wei , Zirui Li , Deze Zeng , Chao Li , and Minyi Guo . 2022 . DVABatch: Diversity-aware Multi-Entry Multi-Exit Batching for Efficient Processing of DNN Services on GPUs. In 2022 USENIX Annual Technical Conference (USENIX ATC 22) . 183--198. Weihao Cui, Han Zhao, Quan Chen, Hao Wei, Zirui Li, Deze Zeng, Chao Li, and Minyi Guo. 2022. DVABatch: Diversity-aware Multi-Entry Multi-Exit Batching for Efficient Processing of DNN Services on GPUs. In 2022 USENIX Annual Technical Conference (USENIX ATC 22). 183--198.
  3. Mingcong Han , Hanze Zhang , Rong Chen , and Haibo Chen . 2022 . Microsecond-scale Preemption for Concurrent {GPU-accelerated}{DNN} Inferences . In 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22) . 539--558. Mingcong Han, Hanze Zhang, Rong Chen, and Haibo Chen. 2022. Microsecond-scale Preemption for Concurrent {GPU-accelerated}{DNN} Inferences. In 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22). 539--558.
  4. Ilya Sutskever , Oriol Vinyals , and Quoc V Le. 2014. Sequence to sequence learning with neural networks. Advances in neural information processing systems 27 ( 2014 ). Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014. Sequence to sequence learning with neural networks. Advances in neural information processing systems 27 (2014).
  5. Jing Wang , Youyou Lu , Qing Wang , Minhui Xie , Keji Huang , and Jiwu Shu . 2022 . Pacman: An Efficient Compaction Approach for {Log-Structured} {Key-Value} Store on Persistent Memory . In 2022 USENIX Annual Technical Conference (USENIX ATC 22) . 773--788. Jing Wang, Youyou Lu, Qing Wang, Minhui Xie, Keji Huang, and Jiwu Shu. 2022. Pacman: An Efficient Compaction Approach for {Log-Structured} {Key-Value} Store on Persistent Memory. In 2022 USENIX Annual Technical Conference (USENIX ATC 22). 773--788.
  6. Gyeong-In Yu , Joo Seong Jeong , Geon-Woo Kim , Soojeong Kim , and Byung-Gon Chun . 2022 . Orca: A Distributed Serving System for {Transformer-Based} Generative Models . In 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22) . 521--538. Gyeong-In Yu, Joo Seong Jeong, Geon-Woo Kim, Soojeong Kim, and Byung-Gon Chun. 2022. Orca: A Distributed Serving System for {Transformer-Based} Generative Models. In 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22). 521--538.
  7. Hong Zhang , Yupeng Tang , Anurag Khandelwal , and Ion Stoica . 2023 . SHEPHERD: Serving DNNs in the Wild. In 20th USENIX Symposium on Networked Systems Design and Implementation (NSDI 23) . USENIX Association, Boston, MA, 787--808. https://www.usenix.org/conference/nsdi23/presentation/zhang-hong Hong Zhang, Yupeng Tang, Anurag Khandelwal, and Ion Stoica. 2023. SHEPHERD: Serving DNNs in the Wild. In 20th USENIX Symposium on Networked Systems Design and Implementation (NSDI 23). USENIX Association, Boston, MA, 787--808. https://www.usenix.org/conference/nsdi23/presentation/zhang-hong
  8. Zhe Zhou , Xuechao Wei , Jiejing Zhang , and Guangyu Sun . 2022 . PetS: A Unified Framework for Parameter-Efficient Transformers Serving. In 2022 USENIX Annual Technical Conference (USENIX ATC 22) . 489--504. Zhe Zhou, Xuechao Wei, Jiejing Zhang, and Guangyu Sun. 2022. PetS: A Unified Framework for Parameter-Efficient Transformers Serving. In 2022 USENIX Annual Technical Conference (USENIX ATC 22). 489--504.

Cited by

See all 1,817

Metadata

Authors
Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez, J., Zhang, H., & Stoica, I.
Affiliations
  • University of California, Berkeley (US)
  • Independent Researcher, Berkeley, United States of America
  • University of California San Diego (US)
Venue
Proceedings of the 29th Symposium on Operating Systems Principles
Publisher
ACM
Published
2023-10-23
Pages
611-626
OpenAlex
W4387321091
Cited by
1,817
References
19
Citations by year
2026: 1017 · 2025: 623 · 2024: 169 · 2023: 8
FWCI
321.20
Type
conference paper
Language
en
Open access
Locations
  • ACM SIGOPS Symposium on Operating Systems Principles (SOSP) (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Funders
  • VMware
  • Zayed University
  • Samsung
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Hardware and Architecture › Parallel Computing and Optimization Techniques
Keywords
large language model serving, memory management, key-value cache, request batching, memory fragmentation, vLLM, inference throughput, dynamic memory allocation

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-09-28, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-09-28 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Efficient Memory Management for Large Language Model Serving with PagedAttention. https://app.cochat.ai/papers/10.1145/3600006.3613165
The paper
Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez, J., Zhang, H., & Stoica, I. (2023). Efficient Memory Management for Large Language Model Serving with PagedAttention. Proceedings of the 29th Symposium on Operating Systems Principles. https://doi.org/10.1145/3600006.3613165

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.