The DOI matches its Crossref record; 60 of the 60 references it cites were found in a registry or an index.
Splitwise: Efficient Generative LLM Inference Using Phase Splitting
Abstract
Generative large language model (LLM) applications are growing rapidly, leading to large-scale deployments of expensive and power-hungry GPUs. Our characterization of LLM inference shows that each inference request undergoes two phases: a compute-intensive prompt computation phase and a memory intensive token generation phase, each with distinct latency, throughput, memory, and power characteristics. Despite state-of-the-art batching and scheduling, the token generation phase underutilizes compute resources. Unlike prompt computation, token generation does not need the compute capability of the latest GPUs and can be run with lower power and cost. Based on these insights, we propose Splitwise, a model deployment and scheduling technique that splits the two phases of LLM inference requests on to separate machines. Splitwise enables phase-specific resource management using hardware that is well suited for each phase. Request state is transferred efficiently between machines using optimized network libraries on the fast back-plane interconnects available in today’s GPU clusters. Using Splitwise, we design homogeneous and heterogeneous LLM inference clusters optimized for throughput, cost, and power Compared to current designs, Splitwise clusters achieve up to $1.4 \times$ higher throughput at $\mathbf{2 0 \%}$ lower cost. Alternatively, they can deliver $2.35 \times$ more throughput under the same power and cost budgets.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Llumnix migrates requests to repair imbalance [Sun et al.(2024)]; DistServe and Splitwise separate prefill and decode placement [Zhong et al.(2024), Patel et al.(2024)].”
From CacheRoute: Planned Prefix-Affinity Routing for Large-Scale LLM Serving (2026) · cites this paper as Patel et al.(2024)
In the citing paper
“The second approach is to separate prefill and decoding to different GPUs to avoid prefilldecoding interference Patel et al. (2024); Zhong et al. (2024).”
From Preble: Efficient Distributed Prompt Scheduling for LLM Serving (2024) · cites this paper as Patel et al. (2024)
In the citing paper
“Several recent works have targeted LLM usages with long prompts and proposed solutions to solve the imbalance problem Agrawal et al. (2023; 2024); Zhong et al. (2024); Patel et al. (2024).”
From Preble: Efficient Distributed Prompt Scheduling for LLM Serving (2024) · cites this paper as Patel et al. (2024)
In the citing paper
“increasing aggregate prefill load and worsening tail latency by intensifying prefill-decode interference in existing LLM serving stacks.”
From PrefillShare: A Shared Prefill Module for KV Reuse in Multi-LLM Disaggregated Serving (2026) · cites this paper as Patel et al., 2024
In the citing paper
“Disaggregated serving reduces such interference by placing prefill and decode on separate GPUs”
From PrefillShare: A Shared Prefill Module for KV Reuse in Multi-LLM Disaggregated Serving (2026) · cites this paper as Patel et al., 2024
In the citing paper
“By isolating prefill execution from the decode loop, disaggregated serving prevents prefill computation from interfering with autoregressive decoding, providing more stable ITL in real-world deployments.”
From PrefillShare: A Shared Prefill Module for KV Reuse in Multi-LLM Disaggregated Serving (2026) · cites this paper as Patel et al., 2024
In the citing paper
“On prefill-only instances [53, 54, 82], lookahead requests consume service time and increase the queuing delay of other requests.”
From SmoothAgent: Efficient Long-Horizon LLM-Based Agent Serving with Lookahead Context Engineering (2026) · cites this paper as [53]
In the citing paper
“Prefill-decode disaggregation [53, 82] eliminates interference between prefill and decode requests by dispatching them to separate instances, and has become the de facto design in large-scale datacenter LLM serving.”
From SmoothAgent: Efficient Long-Horizon LLM-Based Agent Serving with Lookahead Context Engineering (2026) · cites this paper as [53]
In the citing paper
“Prefill-decode (PD) disaggregation places the compute-intensive prefill phase and the memory-intensive decode phase on different nodes, enabling independent resource management [23], [24].”
From Unified AI Gateway: A Framework for Joint Model Routing and KV Cache Management (2026) · cites this paper as [23]
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Jakubův, J., Chvalovský, K., Goertzel, Z., et al. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,648
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325 · cited 2×
“Transformer model sizes have grown steadily, from the early BERT models [36] having 340 million parameters, to GPT-3 [28] with a staggering 175 billion parameters, and GPT-4 rumored to have even more.”
BackgroundLanguage Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · cited 3×
“1 Generative LLM inference for a single request consists of several forward passes through the model, since the output tokens are generated one by one.”
BackgroundInfluentialMapReduce: simplified data processing on large clusters
Dean, J., & Ghemawat, S. (2008) · Communications of the ACM · doi:10.1145/1327452.1327492 · Cited by 18,476 · cited 1×
“Distributed dataflow systems orchestrate large-scale computational graphs and aim to provide general-purpose programmability [34], [46], [75], [82].”
BackgroundRoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515
Transformers: State-of-the-Art Natural Language Processing
Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., et al. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations · doi:10.18653/v1/2020.emnlp-demos.6 · Cited by 8,741
The Hadoop Distributed File System
Shvachko, K. V., Kuang, H., Radia, S., & Chansler, R. J. (2010) · Digest of papers - IEEE Symposium on Mass Storage Systems · doi:10.1109/msst.2010.5496972 · Cited by 4,911 · cited 1×
“Distributed dataflow systems orchestrate large-scale computational graphs and aim to provide general-purpose programmability [34], [46], [75], [82].”
BackgroundResilient distributed datasets: a fault-tolerant abstraction for in-memory cluster computing
Zaharia, M., Chowdhury, M., Das, T., Dave, A., Ma, J. H., McCauley, M., Franklin, M. J. M., Shenker, S. J., & Stoica, I. G. (2012) · Cited by 3,622
Isard, M., Budiu, M., Yu, Y., Birrell, A. D., & Fetterly, D. (2007) · European Conference on Computer Systems · doi:10.1145/1272996.1273005 · Cited by 2,433 · cited 1×
“Distributed dataflow systems orchestrate large-scale computational graphs and aim to provide general-purpose programmability [34], [46], [75], [82].”
Efficient Memory Management for Large Language Model Serving with PagedAttention
Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez, J., Zhang, H., & Stoica, I. (2023) · Proceedings of the 29th Symposium on Operating Systems Principles · doi:10.1145/3600006.3613165 · Cited by 1,817 · cited 5×
“However, once the request is in the token phase, we count it as one active token, since the tokens are generated one at a time (assuming a beam search size of one [51]).”
BackgroundMethodInfluentialEfficient Memory Management for Large Language Model Serving with PagedAttention
Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez, J. E., Zhang, H., & Stoica, I. (2023) · arXiv · doi:10.48550/arxiv.2309.06180 · Cited by 1,802
FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness
Dao, T., Fu, D. Y., Ermon, S., Rudra, A., & Ré, C. (2022) · arXiv · doi:10.48550/arxiv.2205.14135 · Cited by 1,214 · cited 1×
“LLM inference serving is a rapidly developing field, with several recent works optimizing batching [23], [25], [51], [53], [81], scheduling [22], [42], [51], [66], [73], [79], and memory usage [32], [35], [51], [74].”
MethodRay: A Distributed Framework for Emerging AI ApplicationsPreprint
Moritz, P., Nishihara, R., Wang, S., Tumanov, A., Liaw, R., Liang, E., Elibol, M., Yang, Z., Paul, W., Jordan, M. I., et al. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1712.05889 · Cited by 1,040 · cited 1×
“LLM inference under Splitwise can be viewed as a static computational graph with two stages, so it could be implemented using distributed frameworks that provide efficient GPU abstractions [59].”
MethodSchwarzkopf, M., Konwinski, A., Abd-El-Malek, M., & Wilkes, J. (2013) · European Conference on Computer Systems · doi:10.1145/2465351.2465386 · Cited by 651 · cited 1×
“Insights from prior work on partitioned or replicated scheduling could help improve scalability [27], [61], [72] and are orthogonal to Splitwise.”
BackgroundOusterhout, K., Wendell, P., Zaharia, M., & Stoica, I. G. (2013) · ACM SIGOPS Symposium on Operating Systems Principles (SOSP) · doi:10.1145/2517349.2522716 · Cited by 586 · cited 1×
“Insights from prior work on partitioned or replicated scheduling could help improve scalability [27], [61], [72] and are orthogonal to Splitwise.”
BackgroundOpt: Open pre-trained transformer language models
Zhang, S. (2025) · TIB Data Manager · doi:10.57702/dn0pmcbs · Cited by 405
Single-ISA heterogeneous multi-core architectures: the potential for processor power reduction
Kumar, R., Farkas, K. I., Jouppi, N. P., Ranganathan, P., & Tullsen, D. M. (2003) · IEEE/ACM International Symposium on Microarchitecture (MICRO) · doi:10.1109/micro.2003.1253185 · Cited by 373 · cited 1×
“Research on heterogeneous multiprocessor CPU scheduling attempts to match workload heterogeneity to hardware heterogeneity [29], [40], [41], [50], [76], [80].”
BackgroundApollo: scalable and coordinated scheduling for cloud-scale computing
Boutin, É., Ekanayake, J., Lin, W., Shi, B., Zhou, J., Qian, Z., Wu, M., & Zhou, L. (2014) · Operating Systems Design and Implementation · doi:10.5555/2685048.2685071 · Cited by 324
LLM.int8(): 8-Bit Matrix Multiplication for Transformers at Scale
Dettmers, T., Lewis, M., Belkada, Y., & Zettlemoyer, L. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-2198 · Cited by 305
LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale
Dettmers, T., Lewis, M., Belkada, Y., & Zettlemoyer, L. (2022) · arXiv · doi:10.48550/arxiv.2208.07339 · Cited by 305
DeepSpeed- Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scale
Aminabadi, R. Y., Rajbhandari, S., Awan, A. A., Li, C., Li, D., Zheng, E., Ruwase, O., Smith, S., Zhang, M., Rasley, J., et al. (2022) · SC22: International Conference for High Performance Computing, Networking, Storage and Analysis · doi:10.1109/sc41404.2022.00051 · Cited by 297 · cited 1×
“LLM inference serving is a rapidly developing field, with several recent works optimizing batching [23], [25], [51], [53], [81], scheduling [22], [42], [51], [66], [73], [79], and memory usage [32], [35], [51], [74].”
MethodRevisiting network support for RDMA
Mittal, R., Shpiner, A., Panda, A., et al. (2018) · doi:10.1145/3230543.3230557 · Cited by 263 · cited 1×
“The alternative could be HPC clouds, with Infiniband connections through the CPU [3], or Ethernet, using RoCE [58].”
MethodFast Distributed Inference Serving for Large Language ModelsPreprint
Wu, B., Zhong, Y.-M., Zhang, Z.-L., Liu, S.-Y., Liu, F.-Y., Sun, Y.-H., Huang, G., Liu, X.-Z., & Jin, X. (2023) · arXiv.org · doi:10.48550/arxiv.2305.05920 · Cited by 248 · cited 1×
“LLM inference serving is a rapidly developing field, with several recent works optimizing batching [23], [25], [51], [53], [81], scheduling [22], [42], [51], [66], [73], [79], and memory usage [32], [35], [51], [74].”
MethodKwon, Y., Lee, Y., & Rhu, M. (2019) · IEEE/ACM International Symposium on Microarchitecture (MICRO) · doi:10.1145/3352460.3358284 · Cited by 243 · cited 1×
“Prior work exploits such heterogeneity to selectively schedule requests between CPUs and accelerators [38], [52], colocate models with complementary memory usage [30], and partition compute/memory on heterogeneous hardware resources [45], [48].”
BackgroundAnalysis of join-the-shortest-queue routing for web server farms
Gupta, V., Harchol‐Balter, M., Sigman, K., et al. (2007) · Performance Evaluation · doi:10.1016/j.peva.2007.06.012 · Cited by 219 · cited 1×
“CLS uses Join the Shortest Queue (JSQ) scheduling [39], [85] to assign a prompt and a token machine to each request.”
MethodDeepRecSys: A System for Optimizing End-To-End At-Scale Neural Recommendation Inference
Gupta, U., Hsia, S., Saraph, V., Wang, X., Reagen, B., Wei, G.-Y., Lee, H.-H. S., Brooks, D. M., & Wu, C.-J. (2020) · Proceedings - International Symposium on Computer Architecture/Proceedings · doi:10.1109/isca45697.2020.00084 · Cited by 176 · cited 2×
“Prior work exploits such heterogeneity to selectively schedule requests between CPUs and accelerators [38], [52], colocate models with complementary memory usage [30], and partition compute/memory on heterogeneous hardware resources [45], [48].”
BackgroundMArk: Exploiting Cloud Services for Cost-Effective, SLO-Aware Machine Learning Inference Serving.
Zhang, C., Yu, M., Wang, W., & Yan, F. (2019) · Rare & Special e-Zone (The Hong Kong University of Science and Technology) · Cited by 172
Self-citationKannan, R. S., Subramanian, L., Raju, A., Ahn, J., Mars, J., & Tang, L. (2019) · European Conference on Computer Systems · doi:10.1145/3302424.3303958 · Cited by 152 · cited 1×
“Prior work on video and ML serving focuses on scheduling model chains with data dependencies under latency constraints [24], [31], [43], [49], [68].”
BackgroundScheduling heterogeneous multi-cores through performance impact estimation (PIE)
Craeynest, K. V., Jaleel, A., Eeckhout, L., et al. (2012) · Proceedings - International Symposium on Computer Architecture/Proceedings · doi:10.1109/isca.2012.6237019 · Cited by 136
Efficient program scheduling for heterogeneous multi-core processors
Chen, J., & John, L. K. (2009) · Proceedings - ACM IEEE Design Automation Conference · doi:10.1145/1629911.1630149 · Cited by 127 · cited 1×
“Research on heterogeneous multiprocessor CPU scheduling attempts to match workload heterogeneity to hardware heterogeneity [29], [40], [41], [50], [76], [80].”
BackgroundServing DNNs like Clockwork: Performance Predictability from the Bottom UpPreprint
Gujarati, A., Karimi, R. J., Alzayat, S., Hao, W., Kaufmann, A., Vigfússon, Ý., & Mace, J. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.02464 · Cited by 126 · cited 1×
“Since LLM requests are much longer than typical ML requests [37], [38], they incur lower scheduling overhead for similar cluster sizes.”
BackgroundEfficiently Scaling Transformer Inference
Pope, R., Douglas, S., Chowdhery, A., Devlin, J., Bradbury, J., Levskaya, A., Heek, J., Xiao, K., Agrawal, S., & Dean, J. (2022) · arXiv · doi:10.48550/arxiv.2211.05102 · Cited by 113 · cited 1×
“LLM inference serving is a rapidly developing field, with several recent works optimizing batching [23], [25], [51], [53], [81], scheduling [22], [42], [51], [66], [73], [79], and memory usage [32], [35], [51], [74].”
MethodCentaur: A Chiplet-based, Hybrid Sparse-Dense Accelerator for Personalized Recommendations
Hwang, R., Kim, T., Kwon, Y., & Rhu, M. (2020) · Proceedings - International Symposium on Computer Architecture/Proceedings · doi:10.1109/isca45697.2020.00083 · Cited by 109 · cited 1×
“Prior work exploits such heterogeneity to selectively schedule requests between CPUs and accelerators [38], [52], colocate models with complementary memory usage [30], and partition compute/memory on heterogeneous hardware resources [45], [48].”
BackgroundCrankshaw, D., Sela, G.-E., Mo, X., Zumar, C., Stoica, I. G., Gonzalez, J. E., & Tumanov, A. (2020) · ACM Symposium on Cloud Computing (SoCC) · doi:10.1145/3419111.3421285 · Cited by 105 · cited 1×
“Prior work on video and ML serving focuses on scheduling model chains with data dependencies under latency constraints [24], [31], [43], [49], [68].”
BackgroundINFaaS: Automated Model-less Inference Serving
Romero, F., Li, Q., Yadwadkar, N. J., & Kozyrakis, C. (2021) · USENIX Annual Technical Conference · Cited by 102
Characterizing Power Management Opportunities for LLMs in the Cloud
Patel, P., Choukse, E., Zhang, C., Goiri, Í., Warrier, B., Mahalingam, N., & Bianchini, R. (2024) · Proceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 3 · doi:10.1145/3620666.3651329 · Cited by 100 · cited 3×
“Finally, power optimization is attractive for a CSP, since it enables more GPUs to be deployed in the same datacenter [62], [63], but it may not be as important to the user.”
BackgroundMethodSelf-citationThe Datacenter as a Computer: Designing Warehouse-Scale Machines, Third Edition
Barroso, L. A., Hölzle, U., & Ranganathan, P. (2018) · Synthesis lectures on computer architecture · doi:10.2200/s00874ed3v01y201809cac046 · Cited by 93
FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPU
Sheng, Y., Zheng, L., Yuan, B., Li, Z., Ryabinin, M., Fu, D. Y., Xie, Z., Chen, B., Barrett, C., Gonzalez, J. E., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.06865 · Cited by 86 · cited 1×
“LLM inference serving is a rapidly developing field, with several recent works optimizing batching [23], [25], [51], [53], [81], scheduling [22], [42], [51], [66], [73], [79], and memory usage [32], [35], [51], [74].”
MethodSelf-citationDeja Vu: Contextual Sparsity for Efficient LLMs at Inference TimePreprint
Liu, Z., Wang, J., Dao, T., Zhou, T., Yuan, B., Song, Z., Shrivastava, A., Zhang, C., Tian, Y., Ré, C., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2310.17157 · Cited by 76 · cited 1×
“To further reduce our bandwidth utilization, we could also compress the KV-cache before transferring it across the network [55].”
MethodSelf-citationAlpaServe: Statistical Multiplexing with Model Parallelism for Deep Learning ServingPreprint
Li, Z., Zheng, L., Zhong, Y., Liu, V., Sheng, Y., Jin, X., Huang, Y., Chen, Z., Zhang, H., Gonzalez, J. E., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2302.11665 · Cited by 72
Exploiting heterogeneity for tail latency and energy efficiency
Haque, M. E., He, Y., Elnikety, S., Nguyen, T. D., Bianchini, R., & McKinley, K. S. (2017) · IEEE/ACM International Symposium on Microarchitecture (MICRO) · doi:10.1145/3123939.3123956 · Cited by 55 · cited 1×
“Research on heterogeneous multiprocessor CPU scheduling attempts to match workload heterogeneity to hardware heterogeneity [29], [40], [41], [50], [76], [80].”
BackgroundSelf-citationYang, H., Chen, Q., Riaz, M., Luan, Z., Tang, L., & Mars, J. (2017) · Proceedings - International Symposium on Computer Architecture/Proceedings · doi:10.1145/3079856.3080224 · Cited by 47
FleetRec: Large-Scale Recommendation Inference on Hybrid GPU-FPGA Clusters
Jiang, W., He, Z., Zhang, S., et al. (2021) · doi:10.1145/3447548.3467139 · Cited by 36 · cited 1×
“Prior work exploits such heterogeneity to selectively schedule requests between CPUs and accelerators [38], [52], colocate models with complementary memory usage [30], and partition compute/memory on heterogeneous hardware resources [45], [48].”
BackgroundSelf-citationHu, Y., Ghosh, R., & Govindan, R. (2021) · ACM Symposium on Cloud Computing (SoCC) · doi:10.1145/3472883.3486993 · Cited by 35 · cited 1×
“Prior work on video and ML serving focuses on scheduling model chains with data dependencies under latency constraints [24], [31], [43], [49], [68].”
BackgroundFlashDecoding++: Faster Large Language Model Inference on GPUsPreprint
Hong, K., Dai, G., Xu, J., Mao, Q., Li, X., Liu, J., Chen, K., Dong, Y., & Wang, Y. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2311.01282 · Cited by 35 · cited 1×
“LLM inference serving is a rapidly developing field, with several recent works optimizing batching [23], [25], [51], [53], [81], scheduling [22], [42], [51], [66], [73], [79], and memory usage [32], [35], [51], [74].”
MethodHaque, M. E., Eom, Y. H., He, Y., et al. (2015) · ACM SIGPLAN Notices · doi:10.1145/2775054.2694384 · Cited by 29
Self-citationTowards Improved Power Management in Cloud GPUs
Patel, P., Gong, Z., Rizvi, S., et al. (2023) · IEEE Computer Architecture Letters · doi:10.1109/lca.2023.3278652 · Cited by 27 · cited 1×
“This is especially important when building GPU clusters, since GPUs consume much higher power than regular compute machines [63], [64].”
BackgroundSelf-citationSchedTune: A Heterogeneity-Aware GPU Scheduler for Deep Learning
Albahar, H., Dongare, S., Du, Y., Zhao, N., Paul, A. K., & Butt, A. R. (2022) · IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (CCGrid) · doi:10.1109/ccgrid54584.2022.00079 · Cited by 27 · cited 1×
“Prior work on video and ML serving focuses on scheduling model chains with data dependencies under latency constraints [24], [31], [43], [49], [68].”
BackgroundSARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefills
Agrawal, A., Panwar, A., Mohan, J., Kwatra, N., Gulavani, B. S., & Ramjee, R. (2023) · arXiv · doi:10.48550/arxiv.2308.16369 · Cited by 20 · cited 3×
“LLM inference serving is a rapidly developing field, with several recent works optimizing batching [23], [25], [51], [53], [81], scheduling [22], [42], [51], [66], [73], [79], and memory usage [32], [35], [51], [74].”
BackgroundMethodJi, Y., & Gao, S. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.giscience.2023.43 · Cited by 10
POLCA: Power Oversubscription in LLM Cloud ProvidersPreprint
Patel, P., Choukse, E., Zhang, C., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2308.12908 · Cited by 8 · cited 1×
“Finally, power optimization is attractive for a CSP, since it enables more GPUs to be deployed in the same datacenter [62], [63], but it may not be as important to the user.”
MethodSelf-citationAnalysis of JSQ policy on soft real-time scheduling in cluster
Zhu, W. (2000) · doi:10.1109/hpc.2000.846562 · Cited by 2 · cited 1×
“Layer-wise transfer also enables other optimizations, such as earlier start of the token phase in the token machines, as well as earlier release of KV-cache memory on the prompt machines.”
MethodHera: A Heterogeneity-Aware Multi-Tenant Inference Server for Personalized Recommendations
Choi, Y., Kim, J., & Rhu, M. (2025) · International Conference on Parallel Architectures and Compilation Techniques (PACT) · doi:10.1109/pact65351.2025.00037 · Cited by 2 · cited 1×
“Prior work exploits such heterogeneity to selectively schedule requests between CPUs and accelerators [38], [52], colocate models with complementary memory usage [30], and partition compute/memory on heterogeneous hardware resources [45], [48].”
BackgroundHybrid Computing for Interactive Datacenter ApplicationsPreprint
Patel, P., Lim, K., Jhunjhunwalla, K., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2304.04488 · Cited by 1 · cited 1×
“Prior work has studied heterogeneous scheduling for a variety of interactive services [65], [68], [83].”
BackgroundSelf-citationOPT: Open Pre-trained Transformer Language Models
Zhang, S., Roller, S., Goyal, N., Artetxe, M., Chen, M., Chen, S., Dewan, C., Diab, M., Li, X., Lin, X. V., et al. (2022) · arXiv · doi:10.48550/arxiv.2205.01068
BLOOM: A 176B-Parameter Open-Access Multilingual Language Model
Workshop, B., :, Scao, T. L., Fan, A., Akiki, C., Pavlick, E., Ilić, S., Hesslow, D., Castagné, R., Luccioni, A. S., et al. (2022) · arXiv · doi:10.48550/arxiv.2211.05100
Fairness in Serving Large Language Models
Sheng, Y., Cao, S., Li, D., Zhu, B., Li, Z., Zhuo, D., Gonzalez, J. E., & Stoica, I. (2024) · arXiv · doi:10.48550/arxiv.2401.00588 · cited 1×
“LLM inference serving is a rapidly developing field, with several recent works optimizing batching [23], [25], [51], [53], [81], scheduling [22], [42], [51], [66], [73], [79], and memory usage [32], [35], [51], [74].”
MethodHera: A Heterogeneity-Aware Multi-Tenant Inference Server for Personalized Recommendations
Choi, Y., Kim, J., & Rhu, M. (2023) · arXiv · doi:10.48550/arxiv.2302.11750
Research works printed but not linked: 28
- Add Splitwise Implementation to vLLM, GitHub.
- AMD Instinct™ MI250 Accelerator
- Azure InfiniBand HPC VMs
- 2023, Azure Public Dataset: Azure LLM Inference Trace
- CoreWeave - Specialized Cloud Provider
- Google Assistant with Bard
- HPC Interconnect on CoreWeave Cloud
- Intel BigDL-LLM
- Intel Sapphire Rapids with HBM.
- Microsoft Azure ND A100 v4-series
- MSCCL++: A GPU-driven communication stack for scalable AI applications
- Numenta Inference on CPUs
- NVIDIA Accelerated InfiniBand Solutions.
- NVIDIA Chip Shortage
- NVIDIA DGX A100: Universal System for AI Infrastructure.
- NVIDIA DGX H100
- NVIDIA Hopper GPUs Expand Reach as Demand for AI Grows
- OpenAI ChatGPT APIs
- Power Availability Stymies Datacenter Growth
- SplitwiseSim: LLM Serving Cluster Simulator
- The New Bing.
- TurboMind Inference Server
- Patterson, Domain Specific Architectures for Deep Neural Networks: Three Generations of Tensor Processing Units (TPUs), Allen School Distinguished Lecture
- Meta. Introducing the AI Research SuperCluster - Meta’s CuttingEdge AI Supercomputer for AI Research
- Radford, 2019, Language Models are Unsupervised Multitask Learners, OpenAI blog
- Schmid, Fine-tune FLAN-T5 XL/XXL using DeepSpeed & Hugging Face Transformers
- Schmid, Llama 2 is here - Get it on Hugging Face
- Yu, 2022, Orca: A Distributed Serving System for Transformer-Based Generative Models, in OSDI
Cited by
Sorted by citations
EdgeShard: Efficient LLM Inference via Collaborative Edge Computing
Zhang, M., Shen, X., Cao, J., Cui, Z., & Jiang, S. (2024) · IEEE Internet of Things Journal · doi:10.1109/jiot.2024.3524255 · Cited by 297
“This work is distinguished from existing work by partitioning the LLMs and allocating to multiple GPUs in cloud data centers, such as Gpipe [17], PipeDream [18], and Splitwise [19].”
DynamoLLM: Designing LLM Inference Clusters for Performance and Energy Efficiency
Stojkovic, J., Zhang, C., Goiri, Í., Torrellas, J., & Choukse, E. (2025) · 2025 IEEE International Symposium on High Performance Computer Architecture (HPCA) · doi:10.1109/hpca61900.2025.00102 · Cited by 112
“To meet these ever increasing computing demands, researchers have proposed various software [9], [27], [37], [75], [85] and hardware [4], [51], [81] techniques that improve the performance of LLM inference clusters.”
InfluentialSelf-citationTowards Efficient Generative Large Language Model Serving: A Survey from Algorithms to Systems
Miao, X., Oliaro, G., Zhang, Z., Cheng, X., Jin, H., Chen, T., & Jia, Z. (2025) · ACM Computing Surveys · doi:10.1145/3754448 · Cited by 176
“Among the related works, [164] comes closest in subject matter exploring the design of more general Transformer models and domain-specific accelerators.”
Yigit, Y., Ferrag, M. A., Ghanem, M. C., Sarker, I. H., Maglaras, L. A., Chrysoulas, C., Moradpoor, N., Tihanyi, N., & Janicke, H. (2025) · Sensors · doi:10.3390/s25061666 · Cited by 69
Reconciling the contrasting narratives on the environmental impact of large language models
Ren, S., Tomlinson, B., Black, R. W., & Torrance, A. W. (2024) · Scientific Reports · doi:10.1038/s41598-024-76682-6 · Cited by 68
“As a result, the total computational demand of LLM inference can exceed that of training by far 26 , making the amortized training cost for each LLM inference request small in real systems.”
Background
Metadata
- Authors
- Patel, P., Choukse, E., Zhang, C., Shah, A., Goiri, Í., Maleki, S., & Bianchini, R.
- Affiliations
- University of Washington (US)
- Microsoft (United States) (US)
- ORCID
- Venue
- 2024 ACM/IEEE 51st Annual International Symposium on Computer Architecture (ISCA)
- Publisher
- IEEE
- Published
- 2024-06
- Pages
- 118-132
- ISSN
- 1063-6897 (ISSN-L)
- OpenAlex
- W4401211704
- Semantic Scholar
- ad9146d98ae95bbeeef460abe083ecc2c4798672
- Cited by
- 298
- References
- 59
- Citations by year
- 2026: 176 · 2025: 108 · 2024: 14
- FWCI
- 90.63
- Type
- conference paper
- Language
- en
- Open access
- closed
- Licence
- Locations
- Proceedings - International Symposium on Computer Architecture/Proceedings (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Hardware and Architecture › Parallel Computing and Optimization Techniques
- Physical Sciences › Computer Science › Information Systems › Cloud Computing and Resource Management
- Physical Sciences › Computer Science › Computer Networks and Communications › Distributed and Parallel Computing Systems
- Keywords
- large language model inference, phase splitting, token generation, GPU clusters, request scheduling, resource management, power efficiency, cost efficiency, throughput, network libraries
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
