The DOI matches its Crossref record; 60 of the 60 references it cites were found in a registry or an index.
Splitwise: Efficient Generative LLM Inference Using Phase Splitting
Abstract
Generative large language model (LLM) applications are growing rapidly, leading to large-scale deployments of expensive and power-hungry GPUs. Our characterization of LLM inference shows that each inference request undergoes two phases: a compute-intensive prompt computation phase and a memory intensive token generation phase, each with distinct latency, throughput, memory, and power characteristics. Despite state-of-the-art batching and scheduling, the token generation phase underutilizes compute resources. Unlike prompt computation, token generation does not need the compute capability of the latest GPUs and can be run with lower power and cost. Based on these insights, we propose Splitwise, a model deployment and scheduling technique that splits the two phases of LLM inference requests on to separate machines. Splitwise enables phase-specific resource management using hardware that is well suited for each phase. Request state is transferred efficiently between machines using optimized network libraries on the fast back-plane interconnects available in today’s GPU clusters. Using Splitwise, we design homogeneous and heterogeneous LLM inference clusters optimized for throughput, cost, and power Compared to current designs, Splitwise clusters achieve up to $1.4 \times$ higher throughput at $\mathbf{2 0 \%}$ lower cost. Alternatively, they can deliver $2.35 \times$ more throughput under the same power and cost budgets.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Llumnix migrates requests to repair imbalance [Sun et al.(2024)]; DistServe and Splitwise separate prefill and decode placement [Zhong et al.(2024), Patel et al.(2024)].”
From CacheRoute: Planned Prefix-Affinity Routing for Large-Scale LLM Serving (2026) · cites this paper as Patel et al.(2024)
In the citing paper
“The second approach is to separate prefill and decoding to different GPUs to avoid prefilldecoding interference Patel et al. (2024); Zhong et al. (2024).”
From Preble: Efficient Distributed Prompt Scheduling for LLM Serving (2024) · cites this paper as Patel et al. (2024)
In the citing paper
“Several recent works have targeted LLM usages with long prompts and proposed solutions to solve the imbalance problem Agrawal et al. (2023; 2024); Zhong et al. (2024); Patel et al. (2024).”
From Preble: Efficient Distributed Prompt Scheduling for LLM Serving (2024) · cites this paper as Patel et al. (2024)
In the citing paper
“increasing aggregate prefill load and worsening tail latency by intensifying prefill-decode interference in existing LLM serving stacks.”
From PrefillShare: A Shared Prefill Module for KV Reuse in Multi-LLM Disaggregated Serving (2026) · cites this paper as Patel et al., 2024
In the citing paper
“Disaggregated serving reduces such interference by placing prefill and decode on separate GPUs”
From PrefillShare: A Shared Prefill Module for KV Reuse in Multi-LLM Disaggregated Serving (2026) · cites this paper as Patel et al., 2024
In the citing paper
“By isolating prefill execution from the decode loop, disaggregated serving prevents prefill computation from interfering with autoregressive decoding, providing more stable ITL in real-world deployments.”
From PrefillShare: A Shared Prefill Module for KV Reuse in Multi-LLM Disaggregated Serving (2026) · cites this paper as Patel et al., 2024
In the citing paper
“On prefill-only instances [53, 54, 82], lookahead requests consume service time and increase the queuing delay of other requests.”
From SmoothAgent: Efficient Long-Horizon LLM-Based Agent Serving with Lookahead Context Engineering (2026) · cites this paper as [53]
In the citing paper
“Prefill-decode disaggregation [53, 82] eliminates interference between prefill and decode requests by dispatching them to separate instances, and has become the de facto design in large-scale datacenter LLM serving.”
From SmoothAgent: Efficient Long-Horizon LLM-Based Agent Serving with Lookahead Context Engineering (2026) · cites this paper as [53]
In the citing paper
“Prefill-decode (PD) disaggregation places the compute-intensive prefill phase and the memory-intensive decode phase on different nodes, enabling independent resource management [23], [24].”
From Unified AI Gateway: A Framework for Joint Model Routing and KV Cache Management (2026) · cites this paper as [23]
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Jakubův, J., Chvalovský, K., Goertzel, Z., et al. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,648
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325 · cited 2×
“Transformer model sizes have grown steadily, from the early BERT models [36] having 340 million parameters, to GPT-3 [28] with a staggering 175 billion parameters, and GPT-4 rumored to have even more.”
BackgroundLanguage Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · cited 3×
“1 Generative LLM inference for a single request consists of several forward passes through the model, since the output tokens are generated one by one.”
BackgroundInfluentialMapReduce: simplified data processing on large clusters
Dean, J., & Ghemawat, S. (2008) · Communications of the ACM · doi:10.1145/1327452.1327492 · Cited by 18,476 · cited 1×
“Distributed dataflow systems orchestrate large-scale computational graphs and aim to provide general-purpose programmability [34], [46], [75], [82].”
BackgroundRoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515
Transformers: State-of-the-Art Natural Language Processing
Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., et al. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations · doi:10.18653/v1/2020.emnlp-demos.6 · Cited by 8,741
The Hadoop Distributed File System
Shvachko, K. V., Kuang, H., Radia, S., & Chansler, R. J. (2010) · Digest of papers - IEEE Symposium on Mass Storage Systems · doi:10.1109/msst.2010.5496972 · Cited by 4,911 · cited 1×
“Distributed dataflow systems orchestrate large-scale computational graphs and aim to provide general-purpose programmability [34], [46], [75], [82].”
BackgroundResilient distributed datasets: a fault-tolerant abstraction for in-memory cluster computing
Zaharia, M., Chowdhury, M., Das, T., Dave, A., Ma, J. H., McCauley, M., Franklin, M. J. M., Shenker, S. J., & Stoica, I. G. (2012) · Cited by 3,622
Research works printed but not linked: 28
- Add Splitwise Implementation to vLLM, GitHub.
- AMD Instinct™ MI250 Accelerator
- Azure InfiniBand HPC VMs
- 2023, Azure Public Dataset: Azure LLM Inference Trace
- CoreWeave - Specialized Cloud Provider
- Google Assistant with Bard
- HPC Interconnect on CoreWeave Cloud
- Intel BigDL-LLM
- Intel Sapphire Rapids with HBM.
- Microsoft Azure ND A100 v4-series
- MSCCL++: A GPU-driven communication stack for scalable AI applications
- Numenta Inference on CPUs
- NVIDIA Accelerated InfiniBand Solutions.
- NVIDIA Chip Shortage
- NVIDIA DGX A100: Universal System for AI Infrastructure.
- NVIDIA DGX H100
- NVIDIA Hopper GPUs Expand Reach as Demand for AI Grows
- OpenAI ChatGPT APIs
- Power Availability Stymies Datacenter Growth
- SplitwiseSim: LLM Serving Cluster Simulator
- The New Bing.
- TurboMind Inference Server
- Patterson, Domain Specific Architectures for Deep Neural Networks: Three Generations of Tensor Processing Units (TPUs), Allen School Distinguished Lecture
- Meta. Introducing the AI Research SuperCluster - Meta’s CuttingEdge AI Supercomputer for AI Research
- Radford, 2019, Language Models are Unsupervised Multitask Learners, OpenAI blog
- Schmid, Fine-tune FLAN-T5 XL/XXL using DeepSpeed & Hugging Face Transformers
- Schmid, Llama 2 is here - Get it on Hugging Face
- Yu, 2022, Orca: A Distributed Serving System for Transformer-Based Generative Models, in OSDI
Cited by
Sorted by citations
Items 301–309 of 309
Taichi: Elastic Multi-LLM Serving with Coordinated Memory–Compute Multiplexing
Lv, C., Shi, X., Lei, Z., Li, X., Li, X., Qingwen, L., Tang, Z., & Zhao, X. (2026) · IEEE International Conference on Cluster Computing (CLUSTER) · doi:10.1109/cluster68337.2026.00048
LODAR: layered optimization for distributed adaptive LLM routing
Xi, Y., Bing, X., & Yahyapour, R. (2026) · Journal of King Saud University - Computer and Information Sciences · doi:10.1007/s44443-026-01343-9
Semantic Cache Distillation: Efficient State Transfer via Reuse and Selective Patching
Ma, Q., Tang, Z., Cui, H., Yao, Z., & Jia, W. (2026) · arXiv · doi:10.48550/arxiv.2606.07684
“Modern serving systems separate prefill and decode phases across distinct workers to maximize utilization (Zhong et al., 2024; Qin et al., 2024; Patel et al., 2024).”
Li, W., Jiang, G., Ding, X., Tao, Z., Hao, C., Xu, C., Zhang, Y., & Wang, H. (2025) · arXiv · doi:10.48550/arxiv.2504.03775
Distributed LLM Serving on Consumer-Grade GPUs by Reconciling Computation and Communication
Jin, L., Zhang, K., Chen, Y., Zhuoyifan, Li, R., Gao, Y., Yang, B., Cai, Z., & Dong, W. (2025) · Findings of the Association for Computational Linguistics: EMNLP 2025 · doi:10.18653/v1/2025.findings-emnlp.957
Pre-Compiled Pipeline Shards for Distributed LLM Inference on Intel AI PC Fleets
Berenbaum, T., & Venkatachalam, M. (2026) · arXiv · doi:10.48550/arxiv.2608.19147
InferCept: Efficient Intercept Support for Augmented Large Language Model Inference
Abhyankar, R., He, Z., Srivatsa, V., Zhang, H., & Zhang, Y. (2024) · arXiv · doi:10.48550/arxiv.2402.01869
“These metrics are important in meeting service-level objectives and system response time (Zhong et al., 2024; Hu et al., 2024; Patel et al., 2024).”
BackgroundTiwari, S., Chugh, T., Rickert, N., Peter, S., Mahajan, R., & Shen, H. (2026) · arXiv · doi:10.48550/arxiv.2606.16824
Predictive Multi-Tier Memory Management for KV Cache in Large-Scale GPU Inference
Ganjihal, S. R. (2026) · arXiv · doi:10.48550/arxiv.2604.26968
Metadata
- Authors
- Patel, P., Choukse, E., Zhang, C., Shah, A., Goiri, Í., Maleki, S., & Bianchini, R.
- Affiliations
- University of Washington (US)
- Microsoft (United States) (US)
- ORCID
- Venue
- 2024 ACM/IEEE 51st Annual International Symposium on Computer Architecture (ISCA)
- Publisher
- IEEE
- Published
- 2024-06
- Pages
- 118-132
- ISSN
- 1063-6897 (ISSN-L)
- OpenAlex
- W4401211704
- Semantic Scholar
- ad9146d98ae95bbeeef460abe083ecc2c4798672
- Cited by
- 298
- References
- 59
- Citations by year
- 2026: 176 · 2025: 108 · 2024: 14
- FWCI
- 90.63
- Type
- conference paper
- Language
- en
- Open access
- closed
- Licence
- Locations
- Proceedings - International Symposium on Computer Architecture/Proceedings (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Hardware and Architecture › Parallel Computing and Optimization Techniques
- Physical Sciences › Computer Science › Information Systems › Cloud Computing and Resource Management
- Physical Sciences › Computer Science › Computer Networks and Communications › Distributed and Parallel Computing Systems
- Keywords
- large language model inference, phase splitting, token generation, GPU clusters, request scheduling, resource management, power efficiency, cost efficiency, throughput, network libraries
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
