The DOI matches its Crossref record; 60 of the 60 references it cites were found in a registry or an index.
Splitwise: Efficient Generative LLM Inference Using Phase Splitting
Abstract
Generative large language model (LLM) applications are growing rapidly, leading to large-scale deployments of expensive and power-hungry GPUs. Our characterization of LLM inference shows that each inference request undergoes two phases: a compute-intensive prompt computation phase and a memory intensive token generation phase, each with distinct latency, throughput, memory, and power characteristics. Despite state-of-the-art batching and scheduling, the token generation phase underutilizes compute resources. Unlike prompt computation, token generation does not need the compute capability of the latest GPUs and can be run with lower power and cost. Based on these insights, we propose Splitwise, a model deployment and scheduling technique that splits the two phases of LLM inference requests on to separate machines. Splitwise enables phase-specific resource management using hardware that is well suited for each phase. Request state is transferred efficiently between machines using optimized network libraries on the fast back-plane interconnects available in today’s GPU clusters. Using Splitwise, we design homogeneous and heterogeneous LLM inference clusters optimized for throughput, cost, and power Compared to current designs, Splitwise clusters achieve up to $1.4 \times$ higher throughput at $\mathbf{2 0 \%}$ lower cost. Alternatively, they can deliver $2.35 \times$ more throughput under the same power and cost budgets.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Llumnix migrates requests to repair imbalance [Sun et al.(2024)]; DistServe and Splitwise separate prefill and decode placement [Zhong et al.(2024), Patel et al.(2024)].”
From CacheRoute: Planned Prefix-Affinity Routing for Large-Scale LLM Serving (2026) · cites this paper as Patel et al.(2024)
In the citing paper
“The second approach is to separate prefill and decoding to different GPUs to avoid prefilldecoding interference Patel et al. (2024); Zhong et al. (2024).”
From Preble: Efficient Distributed Prompt Scheduling for LLM Serving (2024) · cites this paper as Patel et al. (2024)
In the citing paper
“Several recent works have targeted LLM usages with long prompts and proposed solutions to solve the imbalance problem Agrawal et al. (2023; 2024); Zhong et al. (2024); Patel et al. (2024).”
From Preble: Efficient Distributed Prompt Scheduling for LLM Serving (2024) · cites this paper as Patel et al. (2024)
In the citing paper
“increasing aggregate prefill load and worsening tail latency by intensifying prefill-decode interference in existing LLM serving stacks.”
From PrefillShare: A Shared Prefill Module for KV Reuse in Multi-LLM Disaggregated Serving (2026) · cites this paper as Patel et al., 2024
In the citing paper
“Disaggregated serving reduces such interference by placing prefill and decode on separate GPUs”
From PrefillShare: A Shared Prefill Module for KV Reuse in Multi-LLM Disaggregated Serving (2026) · cites this paper as Patel et al., 2024
In the citing paper
“By isolating prefill execution from the decode loop, disaggregated serving prevents prefill computation from interfering with autoregressive decoding, providing more stable ITL in real-world deployments.”
From PrefillShare: A Shared Prefill Module for KV Reuse in Multi-LLM Disaggregated Serving (2026) · cites this paper as Patel et al., 2024
In the citing paper
“On prefill-only instances [53, 54, 82], lookahead requests consume service time and increase the queuing delay of other requests.”
From SmoothAgent: Efficient Long-Horizon LLM-Based Agent Serving with Lookahead Context Engineering (2026) · cites this paper as [53]
In the citing paper
“Prefill-decode disaggregation [53, 82] eliminates interference between prefill and decode requests by dispatching them to separate instances, and has become the de facto design in large-scale datacenter LLM serving.”
From SmoothAgent: Efficient Long-Horizon LLM-Based Agent Serving with Lookahead Context Engineering (2026) · cites this paper as [53]
In the citing paper
“Prefill-decode (PD) disaggregation places the compute-intensive prefill phase and the memory-intensive decode phase on different nodes, enabling independent resource management [23], [24].”
From Unified AI Gateway: A Framework for Joint Model Routing and KV Cache Management (2026) · cites this paper as [23]
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Jakubův, J., Chvalovský, K., Goertzel, Z., et al. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,648
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325 · cited 2×
“Transformer model sizes have grown steadily, from the early BERT models [36] having 340 million parameters, to GPT-3 [28] with a staggering 175 billion parameters, and GPT-4 rumored to have even more.”
BackgroundLanguage Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · cited 3×
“1 Generative LLM inference for a single request consists of several forward passes through the model, since the output tokens are generated one by one.”
BackgroundInfluentialMapReduce: simplified data processing on large clusters
Dean, J., & Ghemawat, S. (2008) · Communications of the ACM · doi:10.1145/1327452.1327492 · Cited by 18,476 · cited 1×
“Distributed dataflow systems orchestrate large-scale computational graphs and aim to provide general-purpose programmability [34], [46], [75], [82].”
BackgroundRoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515
Transformers: State-of-the-Art Natural Language Processing
Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., et al. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations · doi:10.18653/v1/2020.emnlp-demos.6 · Cited by 8,741
The Hadoop Distributed File System
Shvachko, K. V., Kuang, H., Radia, S., & Chansler, R. J. (2010) · Digest of papers - IEEE Symposium on Mass Storage Systems · doi:10.1109/msst.2010.5496972 · Cited by 4,911 · cited 1×
“Distributed dataflow systems orchestrate large-scale computational graphs and aim to provide general-purpose programmability [34], [46], [75], [82].”
BackgroundResilient distributed datasets: a fault-tolerant abstraction for in-memory cluster computing
Zaharia, M., Chowdhury, M., Das, T., Dave, A., Ma, J. H., McCauley, M., Franklin, M. J. M., Shenker, S. J., & Stoica, I. G. (2012) · Cited by 3,622
Research works printed but not linked: 28
- Add Splitwise Implementation to vLLM, GitHub.
- AMD Instinct™ MI250 Accelerator
- Azure InfiniBand HPC VMs
- 2023, Azure Public Dataset: Azure LLM Inference Trace
- CoreWeave - Specialized Cloud Provider
- Google Assistant with Bard
- HPC Interconnect on CoreWeave Cloud
- Intel BigDL-LLM
- Intel Sapphire Rapids with HBM.
- Microsoft Azure ND A100 v4-series
- MSCCL++: A GPU-driven communication stack for scalable AI applications
- Numenta Inference on CPUs
- NVIDIA Accelerated InfiniBand Solutions.
- NVIDIA Chip Shortage
- NVIDIA DGX A100: Universal System for AI Infrastructure.
- NVIDIA DGX H100
- NVIDIA Hopper GPUs Expand Reach as Demand for AI Grows
- OpenAI ChatGPT APIs
- Power Availability Stymies Datacenter Growth
- SplitwiseSim: LLM Serving Cluster Simulator
- The New Bing.
- TurboMind Inference Server
- Patterson, Domain Specific Architectures for Deep Neural Networks: Three Generations of Tensor Processing Units (TPUs), Allen School Distinguished Lecture
- Meta. Introducing the AI Research SuperCluster - Meta’s CuttingEdge AI Supercomputer for AI Research
- Radford, 2019, Language Models are Unsupervised Multitask Learners, OpenAI blog
- Schmid, Fine-tune FLAN-T5 XL/XXL using DeepSpeed & Hugging Face Transformers
- Schmid, Llama 2 is here - Get it on Hugging Face
- Yu, 2022, Orca: A Distributed Serving System for Transformer-Based Generative Models, in OSDI
Cited by
Sorted by citations
EdgeShard: Efficient LLM Inference via Collaborative Edge Computing
Zhang, M., Shen, X., Cao, J., Cui, Z., & Jiang, S. (2024) · IEEE Internet of Things Journal · doi:10.1109/jiot.2024.3524255 · Cited by 297
“This work is distinguished from existing work by partitioning the LLMs and allocating to multiple GPUs in cloud data centers, such as Gpipe [17], PipeDream [18], and Splitwise [19].”
DynamoLLM: Designing LLM Inference Clusters for Performance and Energy Efficiency
Stojkovic, J., Zhang, C., Goiri, Í., Torrellas, J., & Choukse, E. (2025) · 2025 IEEE International Symposium on High Performance Computer Architecture (HPCA) · doi:10.1109/hpca61900.2025.00102 · Cited by 112
“To meet these ever increasing computing demands, researchers have proposed various software [9], [27], [37], [75], [85] and hardware [4], [51], [81] techniques that improve the performance of LLM inference clusters.”
InfluentialSelf-citationTowards Efficient Generative Large Language Model Serving: A Survey from Algorithms to Systems
Miao, X., Oliaro, G., Zhang, Z., Cheng, X., Jin, H., Chen, T., & Jia, Z. (2025) · ACM Computing Surveys · doi:10.1145/3754448 · Cited by 176
“Among the related works, [164] comes closest in subject matter exploring the design of more general Transformer models and domain-specific accelerators.”
Yigit, Y., Ferrag, M. A., Ghanem, M. C., Sarker, I. H., Maglaras, L. A., Chrysoulas, C., Moradpoor, N., Tihanyi, N., & Janicke, H. (2025) · Sensors · doi:10.3390/s25061666 · Cited by 69
Reconciling the contrasting narratives on the environmental impact of large language models
Ren, S., Tomlinson, B., Black, R. W., & Torrance, A. W. (2024) · Scientific Reports · doi:10.1038/s41598-024-76682-6 · Cited by 68
“As a result, the total computational demand of LLM inference can exceed that of training by far 26 , making the amortized training cost for each LLM inference request small in real systems.”
BackgroundLLMCompass: Enabling Efficient Hardware Design for Large Language Model Inference
Zhang, H., Ning, A., Prabhakar, R. B., & Wentzlaff, D. (2024) · Proceedings - International Symposium on Computer Architecture/Proceedings · doi:10.1109/isca59077.2024.00082 · Cited by 105
“Figure 13 indicates our proposed latency-oriented design and throughput-oriented design are around the sweet point.”
BurstGPT: A Real-World Workload Dataset to Optimize LLM Serving Systems
Wang, Y., Chen, Y., Li, Z., Kang, X., Fang, Y., Zhou, Y., Zheng, Y., Tang, Z., He, X. J., Guo, R., et al. (2025) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3711896.3737413 · Cited by 172
Luo, H., Liu, Y., Zhang, R., Wang, J., Sun, G., Niyato, D. T., Yu, H., Xiong, Z., Wang, X., & Shen, X. (2025) · IEEE Transactions on Cognitive Communications and Networking · doi:10.1109/tccn.2025.3612760 · Cited by 68
“Techniques like split inference, that is, partitioning the model layers between device and cloud, are also explored to cope with this demand, albeit introducing complexity in return [76].”
Accelerating Retrieval-Augmented Generation
Quinn, D., Nouri, M., Patel, N., Salihu, J., Salemi, A., Lee, S., Zamani, H., & Alian, M. (2025) · Proceedings of the 30th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 1 · doi:10.1145/3669940.3707264 · Cited by 48
“Nevertheless, the current state-of-the-art focus in building AI systems is only on accelerating the generation phase [5, 8, 21, 38, 52, 57, 66, 84, 101, 102, 105].”
vAttention: Dynamic Memory Management for Serving LLMs without PagedAttention
Prabhu, R., Nayak, A., Mohan, J., Ramjee, R., & Panwar, A. (2025) · International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) · doi:10.1145/3669940.3707256 · Cited by 42
“This is called the KV cache [36, 63, 78] which accounts for a majority of GPU memory usage during inference.”
InfluentialLLM Inference Serving: Survey of Recent Advances and Opportunities
Li, B., Jiang, Y., Gadepally, V. N., & Tiwari, D. (2024) · IEEE High Performance Extreme Computing Conference · doi:10.1109/hpec62836.2024.10938426 · Cited by 108
“Splitwise [31] extensively characterizes the differences in the execution and utilization patterns of the prefill and decoding stage on different generations of GPUs (heterogeneous hardware).”
Yun, S., Kyung, K., Cho, J., Choi, J., Kim, J., Kim, B., Lee, S., Sohn, K., & Ahn, J. H. (2024) · IEEE/ACM International Symposium on Microarchitecture (MICRO) · doi:10.1109/micro61859.2024.00105 · Cited by 58
“TBT and T2FT are critical performance metrics for LLM inference, especially in conversational tasks [43], which often involve multiple rounds of dialogues between the user and the chatbot.”
InfluentialAchieving Peak Performance for Large Language Models: A Systematic Review
Rostam, Z. R. K., Szénási, S., & Kertész, G. (2024) · IEEE Access · doi:10.1109/access.2024.3424945 · Cited by 46
“LightSeq [63] is a lightweight inference library, addresses the need for efficient and convenient deployment of Trans-former models in online services.”
BackgroundMethodInfluentialCE-CoLLM: Efficient and Adaptive Large Language Models Through Cloud-Edge Collaboration
Jin, H., & Wu, Y. (2025) · IEEE International Conference on Web Services (ICWS) · doi:10.1109/icws67624.2025.00046 · Cited by 39
Helix: Serving Large Language Models over Heterogeneous GPUs and Network via Max-Flow
Mei, Y., Zhuang, Y., Miao, X., Yang, J., Jia, Z., & Vinayak, R. (2025) · Proceedings of the 30th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 1 · doi:10.1145/3669940.3707215 · Cited by 35
“6.3 evaluates the fidelity of the simulator and shows that the simulation errors are less than 5% for all metrics.”
InfluentialTAPAS: Thermal- and Power-Aware Scheduling for LLM Inference in Cloud Platforms
Stojkovic, J., Zhang, C., Goiri, Í., Choukse, E., Qiu, H., Fonseca, R., Torrellas, J., & Bianchini, R. (2025) · International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) · doi:10.1145/3676641.3716025 · Cited by 44
“TAPAS enhances cooling and power oversubscription capabilities while effectively managing infrastructure failures, thereby reducing datacenter TCO with minimal impact on workload performance or result quality.”
InfluentialSelf-citationLi, C., Yin, Y., Wu, X., Zhu, J., Gao, Z., Niu, D., Wu, Q., Si, X., Xie, Y., Zhang, C., et al. (2025) · Proceedings - International Symposium on Computer Architecture/Proceedings · doi:10.1145/3695053.3731008 · Cited by 56
“There are two methods for this problem: (1) Duplicate weights for both prefill and decoding stages like prefill-decoding disaggregation [27, 28, 61, 64, 90].”
Self-citationA Survey: Collaborative Hardware and Software Design in the Era of Large Language Models
Guo, C., Cheng, F., Du, Z., Kiessling, J., Ku, J., Li, S., Li, Z., Ma, M., Molom-Ochir, T., Morris, B., et al. (2025) · IEEE Circuits and Systems Magazine · doi:10.1109/mcas.2024.3476008 · Cited by 32
“SpotServe [88] addresses the challenges of using cloud-based preemptible GPU resources by implementing strategies for dynamic adjustment and state recovery, ensuring robust service Authorized licensed use limited to the terms of the applicable license agreement with IEEE.”
SOFA: A Compute-Memory Optimized Sparsity Accelerator via Cross-Stage Coordinated Tiling
Wang, H., Fang, J., Tang, X., Yue, Z., Li, J., Qin, Y., Guan, S., Yang, Q., Wang, Y., Li, C., et al. (2024) · IEEE/ACM International Symposium on Microarchitecture (MICRO) · doi:10.1109/micro61859.2024.00093 · Cited by 38
“Large language models (LLMs) have driven the trans-former architecture to unprecedented levels of complexity”
Leveraging Large Language Models for Enhancing Safety in Maritime Operations
Miller, T. I., Durlik, I., Kostecka, E., Łobodzińska, A., Łazuga, K., & Kozlovska, P. (2025) · Applied Sciences · doi:10.3390/app15031666 · Cited by 35
Low-Overhead General-Purpose Near-Data Processing in CXL Memory Expanders
Ham, H., Hong, J., Park, G., Shin, Y., Woo, O., Yang, W., Bae, J., Park, E., Sung, H., Lim, E., et al. (2024) · IEEE/ACM International Symposium on Microarchitecture (MICRO) · doi:10.1109/micro61859.2024.00051 · Cited by 33
“Thus, we offload it with NDP, using the output vector of SLS as µ thread pool region.”
Generative AI as a Service in 6G Edge-Cloud: Generation Task Offloading by In-Context Learning
Zhou, H., Hu, C., Yuan, D., Yuan, Y., Wu, D., Liu, X., Han, Z., & Zhang, J. (2024) · IEEE Wireless Communications Letters · doi:10.1109/lwc.2024.3520995 · Cited by 29
“We assume a small-scale LLM is deployed at the network edge as in constraint (4c), in which t ,i , t TTFT,edge , and t TPOT,edge represent total generation time, TTFT, and TPOT values.”
POD-A ttention : Unlocking Full Prefill-Decode Overlap for Faster LLM Inference
Kamath, A. K., Prabhu, R., Mohan, J., Peter, S., Ramjee, R., & Panwar, A. (2025) · Proceedings of the 30th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2 · doi:10.1145/3676641.3715996 · Cited by 26
“Large-scale service providers often depend on expensive high-end GPUs to meet peak demand or latency targets [41].”
InfluentialYao, Z., Tang, Z., Yang, W., & Weijia Jia, B. (2025) · IEEE Transactions on Services Computing · doi:10.1109/tsc.2025.3562362 · Cited by 29
“Patel et al. [37] distribute computation-heavy prompt calculations and memory-heavy token generation across various servers to enhance throughput.”
Sustainable LLM Serving: Environmental Implications, Challenges, and Opportunities : Invited Paper
Ding, Y., & Shi, T. (2024) · International Green and Sustainable Computing Conference · doi:10.1109/igsc64514.2024.00016 · Cited by 24
“LLM serving involves two phases: prefill and decoding [9].”
SpInfer: Leveraging Low-Level Sparsity for Efficient Large Language Model Inference on GPUs
Fan, R., Yu, X., Dong, P., Li, Z., Gong, G., Wang, Q., Wang, W., & Chu, X. (2025) · European Conference on Computer Systems · doi:10.1145/3689031.3717481 · Cited by 34
“Additionally, as inference systems increasingly adopt a de-coupled prefill and decode phase architecture [63, 67, 68, 101], SpInfer’s optimization for the decode phase makes it well-suited for scalable deployment.”
InstAttention: In-Storage Attention Offloading for Cost-Effective Long-Context LLM Inference
Pan, X., Li, E., Li, Q., Liang, S., Shan, Y., Zhou, K., Luo, Y., Wang, X., & Zhang, J. (2025) · IEEE International Symposium on High-Performance Computer Architecture (HPCA) · doi:10.1109/hpca61900.2025.00113 · Cited by 74
“Other solutions [19], [33], [52], [53], [80] leverage disaggregated resources (i.e., GPU, CPU and memory pools) to store the KV cache and accelerate long-context LLM inference, which are not suitable for resource-constrained scenarios.”
ShuffleInfer: Disaggregate LLM Inference for Mixed Downstream Workloads
Hu, C.-C., Huang, H., Xu, L., Chen, X., Wang, C., Xu, J., Chen, S., Feng, H., Wang, S., Bao, Y., et al. (2025) · ACM Transactions on Architecture and Code Optimization · doi:10.1145/3732941 · Cited by 73
“(a) Prefill Schedule: Splitwise [41], Distserve [62], and Dejavu [49] use request batching strategies.”
InfluentialKim, H., Wang, N., Xia, Q., Huang, J., Yazdanbakhsh, A., & Kim, N. S. (2025) · Proceedings - International Symposium on Computer Architecture/Proceedings · doi:10.1145/3695053.3731092 · Cited by 34
Luo, X., Liu, D., Kong, H., Huai, S., Chen, H., Xiong, G., & Liu, W. (2024) · ACM Transactions on Embedded Computing Systems · doi:10.1145/3701728 · Cited by 27
throttLL’eM: Predictive GPU Throttling for Energy Efficient LLM Inference Serving
Kakolyris, A. K., Masouros, D., Vavaroutsos, P., Xydis, S., & Soudris, D. (2025) · IEEE International Symposium on High-Performance Computer Architecture (HPCA) · doi:10.1109/hpca61900.2025.00103 · Cited by 44
“The E2E SLO is set to the p 99 th response time under maximum load.”
InfluentialWindServe: Efficient Phase-Disaggregated LLM Serving with Stream-based Dynamic Scheduling
Feng, J., Huang, Y., Zhang, R., Liang, S., Yan, M., & Wu, J. (2025) · Proceedings of the 52nd Annual International Symposium on Computer Architecture · doi:10.1145/3695053.3730999 · Cited by 16
“In the domain of LLM serving systems, there is a significant body of advancing work focused on enhancing the efficiency and performance of large language model inference [7, 18, 24, 26, 28, 29, 31, 33, 34, 42, 45].”
InfluentialQuality-of-Service Aware LLM Routing for Edge Computing With Multiple Experts
Yang, J., Wu, Q., Feng, Z., Zhou, Z., Guo, D., & Chen, X. (2025) · IEEE Transactions on Mobile Computing · doi:10.1109/tmc.2025.3590969 · Cited by 20
“Some other systems focus on GPU kernel optimization and kernel fusion [21], model parallelism [22], [23], batching algorithm [11], [18], [19], KV-cache management [12], [24] and disaggregated inference [25], [26].”
Performance Characterization of CXL Memory and Its Use Cases
Wang, X., Liu, J., Wu, J., Yang, S., Ren, J., Shankar, B., & Li, D. (2025) · IEEE International Parallel and Distributed Processing Symposium (IPDPS) · doi:10.1109/ipdps64566.2025.00097 · Cited by 16
“LLM can be memory-consuming and have execution phases sensitive to memory band-width [27], [46], [54], hence having potentials to benefit from CXL.”
Liao, J., Chang, F., Xue, Y., Xia, T., Huang, Z., & Wang, Y. (2026) · International Conference on Communications, Information System and Computer Engineering (CISCE) · doi:10.1109/cisce69494.2026.11504731 · Cited by 15
“Splitwise [11] demonstrated the benefits of separating the prefill and decode phases across heterogeneous hardware.”
Yu, X., Jiang, D., Deng, J., et al. (2025) · Proceedings - International Symposium on Computer Architecture/Proceedings · doi:10.1145/3695053.3731016 · Cited by 11
“For larger-scale model training [38, 53], such as GPT-3 or LLaMA, the performance bottleneck is more complex, involving off-die memory bandwidth, inter-chip communication bandwidth, and distributed network communication bandwidth.”
Chimera: Communication Fusion for Hybrid Parallelism in Large Language Models
Qin, L., Cui, J., Cai, W., & Huang, J. (2025) · Proceedings - International Symposium on Computer Architecture/Proceedings · doi:10.1145/3695053.3731025 · Cited by 15
“We adopt hybrid parallelism with the configurations as shown in Due to the autoregressive nature of LLM inference, the process is divided into two phases: prefilling and decoding [48, 76].”
Kim, J., Shin, B., Chung, J., & Rhu, M. (2026) · IEEE International Symposium on High-Performance Computer Architecture (HPCA) · doi:10.1109/hpca68181.2026.11408569 · Cited by 64
“We explore this direction in Section IV-C, where we discuss the implications of serving AI agents over multiple queries with LLM request batching [32], [82], [98].”
An I/O Characterizing Study of Offloading LLM Models and KV Caches to NVMe SSD
Ren, Z., Doekemeijer, K., Matteis, T. D., Pinto, C., Stoica, R., & Trivedi, A. (2025) · Workshop on Challenges and Opportunities of Efficient and Performant Storage Systems (CHEOPS) · doi:10.1145/3719330.3721230 · Cited by 12
“Other products and service names might be trademarks of IBM or other companies.”
Wang, H., Wang, Z., Yue, Z., Long, Y., Wei, T., Yang, J., Wang, Y., Li, C., Wei, S., Hu, Y., et al. (2025) · IEEE/ACM International Symposium on Microarchitecture (MICRO) · doi:10.1145/3725843.3756037 · Cited by 11
A Water Efficiency Dataset for African Data Centers
Shumba, N., Tshekiso, O., Li, P., Fanti, G., & Ren, S. (2025) · Proceedings of the ACM SIGCAS/SIGCHI Conference on Computing and Sustainable Societies · doi:10.1145/3715335.3735483 · Cited by 10
“…the industry practice, since a fully-utilized system can lead to significant SLO violations, which are not tolerable in real-world LLM deployment, especially for commercial LLM applications such as real-time conversations that have strict SLO targets to deliver good quality of experiences [33, 45].”
InfluentialApt-Serve: Adaptive Request Scheduling on Hybrid Cache for Scalable LLM Inference Serving
Gao, S., Zhang, X., Shen, Y., & Chen, L. (2025) · Proceedings of the ACM on Management of Data · doi:10.1145/3725394 · Cited by 17
Dynamic Batch Processing with FlexiDecode Scheduler for Efficient LLM Inference in IIoT
Jia, X., Gu, B., Chen, J., Gao, L., Pang, W., Lv, G. D., Qu, Y., & Cui, L. (2025) · Big Data Mining and Analytics · doi:10.26599/bdma.2025.9020025 · Cited by 9
Understanding the Performance and Power of LLM Inferencing on Edge Accelerators
Arya, M., & Simmhan, Y. L. (2025) · IEEE International Parallel and Distributed Processing Symposium Workshops (IPDPSW) · doi:10.1109/ipdpsw66978.2025.00173 · Cited by 8
ClusterKV: Manipulating LLM KV Cache in Semantic Space for Recallable Compression
Liu, G., Li, C., Zhao, J., Zhang, C., & Guo, M. (2025) · ACM/IEEE Design Automation Conference (DAC) · doi:10.1109/dac63849.2025.11132479 · Cited by 62
“Since the autoregressive decoding stage is typically the performance bottleneck of LLM inference and is memory-bound [8], accessing the KV cache of all previous tokens results in increased latency for long-context inference.”
Self-citationCacheRoute: KV-Cache-Aware Routing for LLM Backend Services on Kubernetes
Pi, S., Zhang, S., & Shawulieti, S. (2026) · International Conference on Electronic Information Technology and Computer Engineering (EITCE) · doi:10.1109/eitce70137.2026.11634378 · Cited by 8
“The separation of prefill and decode execution stages reveals substantial heterogeneity in computational behavior, memory consumption, and latency sensitivity, motivating scheduling mechanisms that operate beyond conventional request-level load balancing [16].”
Zhang, C., You, Y., Wang, N., Park, J., & Zhang, L. (2025) · IEEE Journal on Emerging and Selected Topics in Circuits and Systems · doi:10.1109/jetcas.2025.3575272 · Cited by 13
Self-citationDeploying Foundation Model Powered Agent Services: A Survey
Xu, W., Chen, J., Zheng, P., Yi, X., Tian, T., Zhu, W., Wan, Q., Wang, H., Fan, Y., Su, Q., et al. (2025) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2025.3580745 · Cited by 20
“Reference [254] proposes Splitwise, a technique that schedules the prompt computation and token generation phases to different machines, optimizing hardware utilization and overall efficiency.”
Tan, X., Li, B., Qiu, X., Qu, C., Chu, W., Xu, Y., & Qi, Y. (2025) · International World Wide Web Conference (WWW) · doi:10.1145/3701716.3715247 · Cited by 10
“For complex tasks, LLM-based agents often require lengthy token generation to execute effective planning, which extends TTFT and RT and may hinder usability in time-sensitive applications [19].”
Good things come in small packages: Should we build AI clusters with Lite-GPUs?
Canakci, B., Liu, J., Wu, X., Cheriere, N., Costa, P., Legtchenko, S., Narayanan, D., & Rowstron, A. I. T. (2025) · Workshop on Hot Topics in Operating Systems · doi:10.1145/3713082.3730390 · Cited by 6
“In the evaluation, we assume that different phases can execute on different Lite-GPU clusters [40, 63] to demonstrate the hardware benefits achievable with Lite-GPUs.”
InfluentialELLIE: Energy-Efficient LLM Inference at the Edge Via Prefill-Decode Splitting
Fan, H., Lin, Y.-C., & Prasanna, V. K. (2025) · IEEE International Conference on Application-specific Systems, Architectures and Processors (ASAP) · doi:10.1109/asap65064.2025.00031 · Cited by 8
“Splitwise [13] splits LLM inference into Prefill and Decode stages executed on different machines to improve cost-efficiency, leveraging powerful yet expensive GPUs for the Prefill phase and cost-efficient GPUs for the Decode phase.”
SAGESERVE: Optimizing LLM Serving on Cloud Data Centers with Forecast Aware Auto-Scaling
Jaiswal, S., Jain, K., Simmhan, Y. L., Parayil, A., Mallick, A., Wang, R., Amant, R. S., Bansal, C., Rühle, V., Kulkarni, A., et al. (2025) · Proceedings of the ACM on Measurement and Analysis of Computing Systems · doi:10.1145/3771576 · Cited by 41
“SageServe is validated using a realistic simulation harness that we have developed by extending SplitWise [35], using 4 popular open-source LLMs, for week-long traces from multiple US data centers across two time periods, and also compared against Chiron [36], a State of the Art (SOTA) baseline.”
InfluentialQiu, H., Biswas, A., Zhao, Z., Mohan, J., Khare, A., Choukse, E., Goiri, Í., Zhang, Z., Shen, H., Bansal, C., et al. (2025) · ACM Symposium on Cloud Computing (SoCC) · doi:10.1145/3772052.3772254 · Cited by 6
Self-citationCai, J., Wang, X., Gao, M., Peng, S., Zhu, Z., Wei, Y., Wu, Z., & Ma, K. (2025) · IEEE International Symposium on High-Performance Computer Architecture (HPCA) · doi:10.1109/hpca61900.2025.00048 · Cited by 8
“2) Workload: To comprehensively evaluate SoMa effects and analyze the trade-offs behind different DRAM communication scheduling schemes, we scale batch size from 1 to 64, covering latency-sensitive scenarios to throughput-centric scenarios as introduced in Sec.”
Hermes: Algorithm-System Co-design for Efficient Retrieval-Augmented Generation At-Scale
Shen, M., Umar, M., Maeng, K., Suh, G. E., & Gupta, U. (2025) · Proceedings - International Symposium on Computer Architecture/Proceedings · doi:10.1145/3695053.3731076 · Cited by 8
“Our baseline assumes an input sequence length of 512 tokens and a generated output of 256 tokens, based on an average short length of queries and responses as seen in production systems [35].”
Characterizing and Optimizing LLM Inference Workloads on CPU-GPU Coupled Architectures
Vellaisamy, P., Labonte, T., Chakraborty, S., Turner, M., Sury, S., & Shen, J. P. (2025) · IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS) · doi:10.1109/ispass64960.2025.00015 · Cited by 16
“The prefill stage puts pressure on the compute resources, while the decode stage puts pressure on the memory subsystems [34] [35].”
A Case Study of Environmental Footprints for Generative AI Inference: Cloud versus Edge
Li, P., Islam, M. J., & Ren, S. (2025) · ACM SIGMETRICS Performance Evaluation Review · doi:10.1145/3764944.3764950 · Cited by 5
“When deploying generative AI models in the cloud, further quantization is typically avoided to preserve optimal model performance, especially for models with small and medium sizes [12, 13].”
Liu, M., Gu, J., & Zhao, T. (2025) · IEEE International Conference on Web Services (ICWS) · doi:10.1109/icws67624.2025.00045 · Cited by 5
“Workloads are generated using the Azure LLM Server-less Trace [1], a representative serverless LLM benchmark.”
InfluentialInput-Dependent Power Usage in GPUs
Gregersen, T., Patel, P., & Choukse, E. (2024) · International Conference for High Performance Computing, Networking, Storage and Analysis (SC) · doi:10.1109/scw63240.2024.00235 · Cited by 10
“Research on reducing accelerator peak power draw or energy often focuses on hardware design and the surrounding infrastructure: system design and scheduling [14], [15], [20], [21], power control [9], [22], or efficient chips [23]–[25].”
Self-citationPerformance Characterization of Expert Router for Scalable LLM Inference
Pichlmeier, J., Ross, P., & Luckow, A. (2024) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata62323.2024.10826121 · Cited by 11
“Previous work has investigated how factors such as the number of input tokens, batch size and number of concurrent users can affect these metrics [33]–[35].”
Data+AI: LLM4Data and Data4LLM
Li, G., Wang, J., Zhang, C., & Wang, J. (2025) · ACM SIGMOD International Conference on Management of Data (SIGMOD) · doi:10.1145/3722212.3725641 · Cited by 6
“They propose two-stage disaggregation [25, 44, 45, 69], which splits prefill and decoding computation on different GPUs.”
Self-citationCOMET: Towards Practical W4A4KV4 LLMs Serving
Liu, L., Cheng, L., Ren, H., Zhaohui, X., Pan, Y., Wang, M., Li, X., Han, Y., & Wang, Y. (2025) · International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) · doi:10.1145/3676641.3716252 · Cited by 9
“Explore the performance gains from different optimization strategies.”
Near-Memory LLM Inference Processor based on 3D DRAM-to-logic Hybrid Bonding
Han, S., Yoon, B., Park, G., Song, C., Kim, D., & Kim, J.-J. (2025) · ACM/IEEE Design Automation Conference (DAC) · doi:10.1109/dac63849.2025.11132870 · Cited by 7
“The size of KV cache increases with the batch size linearly so that the batch size is typically limited to less than 64 in real LLM inference servings [25].”
Disaggregated Speculative Decoding for Carbon-Efficient LLM Serving
Shi, T., Wu, Y., Liu, S., & Ding, Y. (2025) · IEEE Computer Architecture Letters · doi:10.1109/lca.2025.3630094 · Cited by 4
“For DPD, we leveraged community implementations of vLLM based on the design principles of Splitwise [1].”
Green prompt engineering for sustainable generative AI
Podder, S., Date, H., & Murthy, S. (2026) · Environmental Science and Ecotechnology · doi:10.1016/j.ese.2026.100684 · Cited by 9
“These two phases tend to be computationally and memory-intensive, respectively [26].”
m2LLM: A Multi-Dimensional Optimization Framework for LLM Inference on Mobile Devices
Liu, K., Zhou, X., & Li, L. (2025) · IEEE Transactions on Parallel and Distributed Systems · doi:10.1109/tpds.2025.3587445 · Cited by 7
“For instance, distributed inference [10], [11], offloading [12], [13], [14], batch optimization [15], [16] have been proposed to optimize the inference process on the cloud environment.”
Efficient LLM Inference via Chunked Prefills
Agrawal, A., Kedia, N., Panwar, A., Mohan, J., Kwatra, N., Gulavani, B. S., Tumanov, A., & Ramjee, R. (2025) · ACM SIGOPS Operating Systems Review · doi:10.1145/3759441.3759444 · Cited by 7
“Similar to Patel et al. [21], to account for the intrinsic performance limitations of a model and hardware pair, we define the SLO on P99 TBT to be equal to 5 x and 25 x the execution time of a decode iteration for a request (with prefilliength of 4k and 32 batch size) running without any prefill…”
Hetis: Serving LLMs in Heterogeneous GPU Clusters with Fine-grained and Dynamic Parallelism
Mo, Z., Liao, J., Xu, H., Zhou, Z., & Xu, C. (2025) · International Conference for High Performance Computing, Networking, Storage and Analysis (SC) · doi:10.1145/3712285.3759784 · Cited by 16
“For instance, the Prefill-Decode disaggregation scheme like Splitwise [35] consumes substantial memory by hosting extra copies of model parameters.”
InfluentialSlower is Greener: Acceptance of Eco-feedback Interventions on Carbon Heavy Internet Services
Kim, H., Young, S., Chen, X., Gupta, U., & Hester, J. D. (2025) · ACM Journal on Computing and Sustainable Societies · doi:10.1145/3723038 · Cited by 4
“Ongoing work also explores designing power and energy efficient servers for salient applications like AI [25, 31].”
Optimizing LLM Inference: Fluid-Guided Online Scheduling with Memory ConstraintsPreprint
Ao, R., Luo, G., Simchi‐Levi, D., & Wang, X. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5195463 · Cited by 4
BATON: Enhancing Batch-wise Inference Efficiency for Large Language Models via Dynamic Re-batching
Cong, P., Chen, Q., Zhao, H., & Yang, T. (2025) · International World Wide Web Conference (WWW) · doi:10.1145/3696410.3714950 · Cited by 7
“Moreover, the inference process can be divided into two phases: prefilling and decoding [15, 16].”
Cannikin: No Lagger of SLO in Concurrent Multiple LoRA LLM Serving
Zhu, R., Jiang, Z., Zhang, Z., Liu, X., Liu, X., & Jin, X. (2025) · IEEE Transactions on Parallel and Distributed Systems · doi:10.1109/tpds.2025.3590014 · Cited by 7
“Distserve [15] and SplitWise [16] disaggregate prefill phases and decoding phases into different instances to improve the system efficiency and cost.”
InfluentialS2M3: Split-and-Share Multi-Modal Models for Distributed Multi-Task Inference on the Edge
Yoon, J., Lee, J., He, T., Choi, N., & Ji, B. (2025) · IEEE International Conference on Distributed Computing Systems (ICDCS) · doi:10.1109/icdcs63083.2025.00089 · Cited by 4
“…✓ ✗ LLM-Pruner [8], SparseGPT [9], LGViT [33], PowerInfer [34] Inference Lightweight Model ✗ ✗ DINA [16], A3C [17], PETALS [18], Splitwise [19], EdgeShard [20] Inference Intra-Module Partitioning ✗ ✗ We design S2M3 , a S plit-and-S hare M ulti-M odal M odel architecture for multi-task…”
LibraPIM: Dynamic Load Rebalancing to Maximize Utilization in PIM-Assisted LLM Inference Systems
Cho, H., Jang, Y., Kim, H., et al. (2025) · International Conference on Parallel Architectures and Compilation Techniques · doi:10.1109/pact65351.2025.00016 · Cited by 3
“LLM inference typically relies on compute accelerators, such as GPUs [11, 12] and NPUs [13, 14, 15, 16, 17, 18, 19, 20], which exploit massive parallelism to accelerate computationally intensive layers.”
BucketServe: Bucket-Based Dynamic Batching for Smart and Efficient LLM Inference Serving
Zheng, W., Xu, M., Song, S. H., & Ye, K. (2025) · IEEE International Conference on Internet of Things (iThings) and IEEE Green Computing and Communications (GreenCom) and IEEE Cyber, Physical and Social Computing (CPSCom) and IEEE Smart Data (SmartData) · doi:10.1109/ithings-greencom-cpscom-smartdata-cybermatics67059.2025.00041 · Cited by 3
Zheng, T., Bi, Y., Han, G., et al. (2026) · IEEE Transactions on Mobile Computing · doi:10.1109/tmc.2026.3650838 · Cited by 3
“[ 27] introduces Splitwise, a model deployment and scheduling method that splits the prefill and decoding phases across separate devices.”
Sometimes Painful but Promising: Feasibility and Trade-Offs of On-Device Language Model Inference
Abstreiter, M., Tarkoma, S., & Morabito, R. (2026) · ACM Transactions on Embedded Computing Systems · doi:10.1145/3788870 · Cited by 12
“Despite having 8 GB of nominal memory, the usable memory was limited to below 7 GB, as the operating system and background processes collectively occupied over 1 GB.”
GPU-Centric Memory Tiering for LLM Serving With NVIDIA Grace Hopper Superchip
Choi, W.-H., Jeong, J., Jang, H., & Ahn, J. (2025) · IEEE Computer Architecture Letters · doi:10.1109/lca.2025.3533588 · Cited by 3
Topology-Aware Virtualization over Inter-Core Connected Neural Processing Units
Feng, D., Feng, E., Du, D., Xu, P., Xia, Y., Chen, H., & Zhao, R. J. (2025) · Proceedings - International Symposium on Computer Architecture/Proceedings · doi:10.1145/3695053.3731002 · Cited by 3
FlexNeRFer: A Multi-Dataflow, Adaptive Sparsity-Aware Accelerator for On-Device NeRF Rendering
Noh, S.-H., Shin, B., Choi, J., et al. (2025) · Proceedings - International Symposium on Computer Architecture/Proceedings · doi:10.1145/3695053.3731107 · Cited by 3
“As the demand for Transformer-based LLMs continues to grow across various real-world applications, research efforts in both academia and industry have been actively directed toward improving inference speed [20, 23, 30, 43, 56, 57, 63, 66, 73, 97].”
Chameleon: Adaptive Caching and Scheduling for Many-Adapter LLM Inference Environments
Iliakopoulou, N., Stojkovic, J., Alverti, C., Xu, T., Franke, H., & Torrellas, J. (2025) · IEEE/ACM International Symposium on Microarchitecture (MICRO) · doi:10.1145/3725843.3756083 · Cited by 15
Engine-Agnostic Model Hot-Swapping for Cost-Effective LLM Inference
Stoyanov, R., Spišaková, V., Reber, A., Armour, W., Copik, M., & Bruno, R. (2025) · International Conference for High Performance Computing, Networking, Storage and Analysis (SC) · doi:10.1145/3731599.3767354 · Cited by 2
“This overhead far exceeds the latency requirements for token generation, with industry target for high-quality interactive experience typically below the average human reaction time in the range of 200-250 ms [41].”
Guo, T., Zhang, X.-W., Du, J.-S., Chen, Z.-G., Xiao, N., & Lu, Y.-T. (2025) · International Conference on Software Composition · doi:10.1145/3712285.3759823 · Cited by 15
“Finally, a sampling strategy (e.g., greedy, top-k, or nucleus sampling) is applied to select the next token for generation.”
Tropical: Enhancing SLO Attainment in Disaggregated LLM Serving via SLO-Aware Multiplexing
Ma, J., Chen, J., Li, X., Duan, J., Duanmu, H., Zhang, X., Yang, C., & Lin, D. (2025) · ACM/IEEE Design Automation Conference (DAC) · doi:10.1109/dac63849.2025.11132617 · Cited by 2
“Similar conclusions can also be drawn from the dataset evaluated in other LLM serving-related works [7].”
A Deployment-Aware Framework for Carbon- and Water- Efficient LLM Serving
Hoxha, J., Thanasi-Boçe, M., & Khalifa, T. (2025) · Sustainability · doi:10.3390/su172310473 · Cited by 3
M etis : A Non-Clairvoyant, Workflow-Aware OS Scheduler for Serverless Applications
Tang, W., Yang, Y., & Wu, J. (2025) · ACM Symposium on Cloud Computing (SoCC) · doi:10.1145/3772052.3772221 · Cited by 2
“These serverless work-flows, represented as directed acyclic graphs (DAGs), have gained significant adoption across diverse domains including video processing pipelines [3, 53], machine learning inference chains [16, 37], and large-scale data analytics [40].”
Oneiros: KV Cache Optimization through Parameter Remapping for Multi-tenant LLM Serving
Li, R., Pal, S., Pullu, V. N., Sinha, P., Ryoo, J., John, L. K., & Yadwadkar, N. J. (2025) · ACM Symposium on Cloud Computing (SoCC) · doi:10.1145/3772052.3772215 · Cited by 5
“Traces: We evaluate Oneiros using the Azure coding LLM inference traces that capture real-world query arrival patterns characterized by bursty query patterns [46, 60].”
InfluentialAdaptive Request Scheduling and Load Balancing for Edge Deployed Large Language Models
Mou, F., Tang, Z., Weijia Jia, B., & Zhao, W. (2026) · IEEE Transactions on Services Computing · doi:10.1109/tsc.2026.3665250 · Cited by 2
“Recent resource-aware edge inference systems have explored distributed LLM serving at the edge [19], [20].”
RPU – A Reasoning Processing Unit
Adiletta, M. J., Wei, G.-Y., & Brooks, D. M. (2026) · IEEE International Symposium on High-Performance Computer Architecture (HPCA) · doi:10.1109/hpca68181.2026.11408490 · Cited by 2
“The first step toward low-latency LLM inference is to separate prefill and decode onto different systems, as in Dynamo [17] and the Splitwise execution model [50].”
VectorLiteRAG: Latency-Aware and Fine-Grained Resource Partitioning for Efficient RAG
Kim, J., & Mahajan, D. (2026) · IEEE International Symposium on High-Performance Computer Architecture (HPCA) · doi:10.1109/hpca68181.2026.11408552 · Cited by 9
“This allows in-stantiating a Beta distribution with inferred mean and variance for any cache coverage configuration.”
AUM: Unleashing the Efficiency Potential of Shared Processors with Accelerator Units for LLM Serving
Wang, X., Li, C., Zhuansun, Y., Guo, J., Hou, X., Wang, J., Wang, L., Chen, W., Huang, C., Yang, G., et al. (2026) · IEEE International Symposium on High-Performance Computer Architecture (HPCA) · doi:10.1109/hpca68181.2026.11408539 · Cited by 2
“Firstly, AU-enabled CPU should be used as prompt machines under specific scenarios [67].”
InfluentialBlitzScale: Fast and Live Large Model Autoscaling with O(1) Host Caching
Zhang, D., Liu, Y., Wang, H., Wei, X., Shan, Y., Chen, R., & Chen, H. Z. (2026) · ACM Transactions on Computer Systems · doi:10.1145/3807606 · Cited by 2
“Because the scaling requirements are closely related to the incoming request rates, we chose three typical real-world traces: BurstGPT [71], Azure-Code and AzureConv, both from Azure [55].”
InfluentialSLICE: S LO-Driven Scheduling for L LM I nference on E dge Computing Devices
Zhou, P., Lei, Y., Liu, L., Xu, X., Yu, H., Sun, G., Ergu, D., & Dai, Y. (2026) · IEEE Transactions on Services Computing · doi:10.1109/tsc.2026.3699152 · Cited by 2
“Splitwise[32]decoupledprefillingand decodingbatchestoreduceinterference.”
Wei, Y., Cai, J., Gao, M., et al. (2025) · Proceedings - ACM IEEE Design Automation Conference · doi:10.1109/dac63849.2025.11132440 · Cited by 2
“To further investigate the architecture and layer-pipeline mapping of many-core accelerators, we conduct a DSE, providing in-depth analysis and discussion on the two critical factors of mapping and architecture: pipeline depth and buffer size.”
LLMShare: Optimizing LLM Inference Serving with Hardware Architecture Exploration
Liu, H., Bai, C., Xu, P., Yin, L., Yu, X. D., Zhen, H., Yuan, M., Ho, T.-Y., & Bei, Y. (2025) · ACM/IEEE Design Automation Conference (DAC) · doi:10.1109/dac63849.2025.11132534 · Cited by 2
“For instance, the prefill stage is computation-intensive and requires limited batch sizes to maintain performance, whereas the decoding phase can handle larger batch sizes, enabling higher throughput [17].”
InfluentialLiakopoulos, D., Sinha, P., Hu, T., Lee, M., & Yadwadkar, N. J. (2025) · International Conference for High Performance Computing, Networking, Storage and Analysis (SC) · doi:10.1145/3712285.3759867 · Cited by 2
“Several works focus on optimizing request scheduling [2, 3, 7, 22, 55, 67, 81, 88, 104, 107], reconfiguration [94], and speculative execution [56, 98].”
InfluentialZhang, Z., Shen, H., Vargaftik, S., Basat, R. B., Mitzenmacher, M., & Yu, M. (2025) · Proceedings of the ACM SIGCOMM 2025 Conference · doi:10.1145/3718958.3750481 · Cited by 2
DroidSpeak: KV Cache Sharing for Cross-LLM Communication and Multi-LLM Serving
Liu, Y., Huang, Y., Yao, J., Feng, S., Gu, Z., Du, K., Li, H., Cheng, Y., Jiang, J., Lu, S., et al. (2024) · arXiv · doi:10.48550/arxiv.2411.02820 · Cited by 2
Self-citationPreble: Efficient Distributed Prompt Scheduling for LLM Serving
Srivatsa, V., He, Z., Abhyankar, R., Li, D., & Zhang, Y. (2024) · arXiv · doi:10.48550/arxiv.2407.00023 · Cited by 1
“It supports both data parallelism and model parallelism.”
BackgroundMethodOnline Scheduling for LLM Inference with KV Cache Constraints
Jaillet, P., Jiang, J., Mellou, K., Molinaro, M., Podimata, C., & Zhou, Z. (2025) · arXiv · doi:10.48550/arxiv.2502.07115 · Cited by 1
“While many studies have focused on improving LLM inference efficiency via engineering [Agrawal et al., 2023, Kwon et al., 2023, Patel et al., 2023, Pope et al., 2023, Sheng et al., 2023, Yu et al., 2022, Zhong et al., 2024], there are very few formal models in this space.”
Metadata
- Authors
- Patel, P., Choukse, E., Zhang, C., Shah, A., Goiri, Í., Maleki, S., & Bianchini, R.
- Affiliations
- University of Washington (US)
- Microsoft (United States) (US)
- ORCID
- Venue
- 2024 ACM/IEEE 51st Annual International Symposium on Computer Architecture (ISCA)
- Publisher
- IEEE
- Published
- 2024-06
- Pages
- 118-132
- ISSN
- 1063-6897 (ISSN-L)
- OpenAlex
- W4401211704
- Semantic Scholar
- ad9146d98ae95bbeeef460abe083ecc2c4798672
- Cited by
- 298
- References
- 59
- Citations by year
- 2026: 176 · 2025: 108 · 2024: 14
- FWCI
- 90.63
- Type
- conference paper
- Language
- en
- Open access
- closed
- Licence
- Locations
- Proceedings - International Symposium on Computer Architecture/Proceedings (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Hardware and Architecture › Parallel Computing and Optimization Techniques
- Physical Sciences › Computer Science › Information Systems › Cloud Computing and Resource Management
- Physical Sciences › Computer Science › Computer Networks and Communications › Distributed and Parallel Computing Systems
- Keywords
- large language model inference, phase splitting, token generation, GPU clusters, request scheduling, resource management, power efficiency, cost efficiency, throughput, network libraries
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
