The DOI matches its Crossref record; 60 of the 60 references it cites were found in a registry or an index.
Splitwise: Efficient Generative LLM Inference Using Phase Splitting
Abstract
Generative large language model (LLM) applications are growing rapidly, leading to large-scale deployments of expensive and power-hungry GPUs. Our characterization of LLM inference shows that each inference request undergoes two phases: a compute-intensive prompt computation phase and a memory intensive token generation phase, each with distinct latency, throughput, memory, and power characteristics. Despite state-of-the-art batching and scheduling, the token generation phase underutilizes compute resources. Unlike prompt computation, token generation does not need the compute capability of the latest GPUs and can be run with lower power and cost. Based on these insights, we propose Splitwise, a model deployment and scheduling technique that splits the two phases of LLM inference requests on to separate machines. Splitwise enables phase-specific resource management using hardware that is well suited for each phase. Request state is transferred efficiently between machines using optimized network libraries on the fast back-plane interconnects available in today’s GPU clusters. Using Splitwise, we design homogeneous and heterogeneous LLM inference clusters optimized for throughput, cost, and power Compared to current designs, Splitwise clusters achieve up to $1.4 \times$ higher throughput at $\mathbf{2 0 \%}$ lower cost. Alternatively, they can deliver $2.35 \times$ more throughput under the same power and cost budgets.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Llumnix migrates requests to repair imbalance [Sun et al.(2024)]; DistServe and Splitwise separate prefill and decode placement [Zhong et al.(2024), Patel et al.(2024)].”
From CacheRoute: Planned Prefix-Affinity Routing for Large-Scale LLM Serving (2026) · cites this paper as Patel et al.(2024)
In the citing paper
“The second approach is to separate prefill and decoding to different GPUs to avoid prefilldecoding interference Patel et al. (2024); Zhong et al. (2024).”
From Preble: Efficient Distributed Prompt Scheduling for LLM Serving (2024) · cites this paper as Patel et al. (2024)
In the citing paper
“Several recent works have targeted LLM usages with long prompts and proposed solutions to solve the imbalance problem Agrawal et al. (2023; 2024); Zhong et al. (2024); Patel et al. (2024).”
From Preble: Efficient Distributed Prompt Scheduling for LLM Serving (2024) · cites this paper as Patel et al. (2024)
In the citing paper
“increasing aggregate prefill load and worsening tail latency by intensifying prefill-decode interference in existing LLM serving stacks.”
From PrefillShare: A Shared Prefill Module for KV Reuse in Multi-LLM Disaggregated Serving (2026) · cites this paper as Patel et al., 2024
In the citing paper
“Disaggregated serving reduces such interference by placing prefill and decode on separate GPUs”
From PrefillShare: A Shared Prefill Module for KV Reuse in Multi-LLM Disaggregated Serving (2026) · cites this paper as Patel et al., 2024
In the citing paper
“By isolating prefill execution from the decode loop, disaggregated serving prevents prefill computation from interfering with autoregressive decoding, providing more stable ITL in real-world deployments.”
From PrefillShare: A Shared Prefill Module for KV Reuse in Multi-LLM Disaggregated Serving (2026) · cites this paper as Patel et al., 2024
In the citing paper
“On prefill-only instances [53, 54, 82], lookahead requests consume service time and increase the queuing delay of other requests.”
From SmoothAgent: Efficient Long-Horizon LLM-Based Agent Serving with Lookahead Context Engineering (2026) · cites this paper as [53]
In the citing paper
“Prefill-decode disaggregation [53, 82] eliminates interference between prefill and decode requests by dispatching them to separate instances, and has become the de facto design in large-scale datacenter LLM serving.”
From SmoothAgent: Efficient Long-Horizon LLM-Based Agent Serving with Lookahead Context Engineering (2026) · cites this paper as [53]
In the citing paper
“Prefill-decode (PD) disaggregation places the compute-intensive prefill phase and the memory-intensive decode phase on different nodes, enabling independent resource management [23], [24].”
From Unified AI Gateway: A Framework for Joint Model Routing and KV Cache Management (2026) · cites this paper as [23]
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Jakubův, J., Chvalovský, K., Goertzel, Z., et al. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,648
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325 · cited 2×
“Transformer model sizes have grown steadily, from the early BERT models [36] having 340 million parameters, to GPT-3 [28] with a staggering 175 billion parameters, and GPT-4 rumored to have even more.”
BackgroundLanguage Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · cited 3×
“1 Generative LLM inference for a single request consists of several forward passes through the model, since the output tokens are generated one by one.”
BackgroundInfluentialMapReduce: simplified data processing on large clusters
Dean, J., & Ghemawat, S. (2008) · Communications of the ACM · doi:10.1145/1327452.1327492 · Cited by 18,476 · cited 1×
“Distributed dataflow systems orchestrate large-scale computational graphs and aim to provide general-purpose programmability [34], [46], [75], [82].”
BackgroundRoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515
Transformers: State-of-the-Art Natural Language Processing
Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., et al. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations · doi:10.18653/v1/2020.emnlp-demos.6 · Cited by 8,741
The Hadoop Distributed File System
Shvachko, K. V., Kuang, H., Radia, S., & Chansler, R. J. (2010) · Digest of papers - IEEE Symposium on Mass Storage Systems · doi:10.1109/msst.2010.5496972 · Cited by 4,911 · cited 1×
“Distributed dataflow systems orchestrate large-scale computational graphs and aim to provide general-purpose programmability [34], [46], [75], [82].”
BackgroundResilient distributed datasets: a fault-tolerant abstraction for in-memory cluster computing
Zaharia, M., Chowdhury, M., Das, T., Dave, A., Ma, J. H., McCauley, M., Franklin, M. J. M., Shenker, S. J., & Stoica, I. G. (2012) · Cited by 3,622
Research works printed but not linked: 28
- Add Splitwise Implementation to vLLM, GitHub.
- AMD Instinct™ MI250 Accelerator
- Azure InfiniBand HPC VMs
- 2023, Azure Public Dataset: Azure LLM Inference Trace
- CoreWeave - Specialized Cloud Provider
- Google Assistant with Bard
- HPC Interconnect on CoreWeave Cloud
- Intel BigDL-LLM
- Intel Sapphire Rapids with HBM.
- Microsoft Azure ND A100 v4-series
- MSCCL++: A GPU-driven communication stack for scalable AI applications
- Numenta Inference on CPUs
- NVIDIA Accelerated InfiniBand Solutions.
- NVIDIA Chip Shortage
- NVIDIA DGX A100: Universal System for AI Infrastructure.
- NVIDIA DGX H100
- NVIDIA Hopper GPUs Expand Reach as Demand for AI Grows
- OpenAI ChatGPT APIs
- Power Availability Stymies Datacenter Growth
- SplitwiseSim: LLM Serving Cluster Simulator
- The New Bing.
- TurboMind Inference Server
- Patterson, Domain Specific Architectures for Deep Neural Networks: Three Generations of Tensor Processing Units (TPUs), Allen School Distinguished Lecture
- Meta. Introducing the AI Research SuperCluster - Meta’s CuttingEdge AI Supercomputer for AI Research
- Radford, 2019, Language Models are Unsupervised Multitask Learners, OpenAI blog
- Schmid, Fine-tune FLAN-T5 XL/XXL using DeepSpeed & Hugging Face Transformers
- Schmid, Llama 2 is here - Get it on Hugging Face
- Yu, 2022, Orca: A Distributed Serving System for Transformer-Based Generative Models, in OSDI
Cited by
Sorted by citations
Items 101–200 of 309
Adaptive Batch Budget for LLM Inference
Yeşil, Ç., Ay, B. T., Ak, F. A., Mercan, Ö. B., & Nefesoğlu, O. (2024) · International Conference on Computer Science and Engineering (UBMK) · doi:10.1109/ubmk63289.2024.10773573 · Cited by 1
Throughput-Oriented LLM Inference via KV-Activation Hybrid Caching with A Single GPU
Lee, S., Kim, H., Hwang, S., Heo, G., Noh, M., & Huh, J. (2025) · IEEE International Conference on Computer Design: VLSI in Computers and Processors (ICCD) · doi:10.1109/iccd65941.2025.00045 · Cited by 5
“Approaches in this domain are designed to sustain high throughput under strict latency constraints [3], [4], [10]–[12], despite the difference of main ideas.”
DynamicAttention: Dynamic KV Cache for Disaggregate LLM Inference
Ding, Z., & Yang, T. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10890367 · Cited by 1
Improving the Efficiency of LLM Inference Serving Systems
Papaioannou, K., & Doudali, T. D. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-90203-1_39 · Cited by 1
Chip Architectures Under Advanced Computing Sanctions
Ning, A., & Wentzlaff, D. (2025) · Proceedings - International Symposium on Computer Architecture/Proceedings · doi:10.1145/3695053.3731012 · Cited by 1
AiDE: Attention-FFN Disaggregated Execution for Cost-Effective LLM Decoding on CXL-PNM
Kim, K.-S., Kwon, O., Park, Y., & Lee, J. W. (2025) · IEEE Computer Architecture Letters · doi:10.1109/lca.2025.3597323 · Cited by 1
Cost-Efficient VM Selection for Cloud-Based LLM Inference with KV Cache Offloading
Kim, K., Kim, J., Chung, H., Cha, M.-H., Kim, H.-Y., & Kim, Y. (2025) · IEEE ... International Conference on Cloud Computing · doi:10.1109/cloud67622.2025.00027 · Cited by 1
RTT- or Bandwidth-Bound? Demystifying the KV Cache Transfer in Large Language Model Serving
Yue, S., Wang, M., Yan, Y., Cheng, W., Jiang, Z., & Zhang, Z. (2025) · doi:10.1145/3748273.3749196 · Cited by 1
He, Y., Xu, M., Wu, J., Hu, J., Ma, C., Shen, M. X., Chen, L., Xu, C., Qu, L., & Ke-jiang, Y. (2026) · Software Practice and Experience · doi:10.1002/spe.70054 · Cited by 5
Efficient KV Cache Spillover Management on Memory-Constrained GPU for LLM Inference
Jiang, J., Chen, Y., Zhang, Z., He, B., Luo, P., Lu, M., Chen, Y., Zhang, H., Du, J., Huang, D., et al. (2025) · IEEE Transactions on Parallel and Distributed Systems · doi:10.1109/tpds.2025.3626974 · Cited by 2
AsymServe: Demystifying and Optimizing LLM Serving Efficiency on CPU Acceleration Units
Wang, X., Zhuansun, Y., Li, C., Wang, J., Hou, X., Sun, L., Wang, L., & Guo, M. (2025) · Lecture notes in computer science · doi:10.1007/978-981-95-1021-4_17 · Cited by 1
Your network doesn't end at the NIC
Joshi, R., Agarwal, S., Lao, C., & Yu, M. (2025) · ACM Workshop on Hot Topics in Networks (HotNets) · doi:10.1145/3772356.3772415 · Cited by 1
TransLock: Securing LLM deployment for software applications via self-locking watermarks
Ye, P.-G., Zhang, Z., Mo, K., & Peng, Y. (2025) · Empirical Software Engineering · doi:10.1007/s10664-025-10750-x · Cited by 1
Nova: Real-Time Agentic Vision-Language Model Serving With Adaptive Cross-Stage Parallelization
Xu, Y., Liu, S., Zhang, D., Yan, B., Wu, F., & Chen, G. (2025) · IEEE Real-Time Systems Symposium (RTSS) · doi:10.1109/rtss66672.2025.00038 · Cited by 2
Laser: Unlocking Layer-Level Scheduling for Efficient Multi-SLO LLM Serving
Liao, J., Dong, Q., Liang, Y., Zhou, Z., & Chen, X. (2026) · ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming · doi:10.1145/3774934.3786413 · Cited by 5
DOPD: A Dynamic PD-Disaggregation Architecture for Maximizing Goodput in LLM Inference Serving
Liao, J., Xu, M., Zheng, W., Wang, Y., Ye, K., Buyya, R., & Xu, C. (2026) · IEEE Transactions on Services Computing · doi:10.1109/tsc.2026.3670011 · Cited by 1
Adaptive Draft Sequence Length: Enhancing Speculative Decoding Throughput on PIM-Enabled Systems
Wang, R., Wang, Q., Liu, H., Zheng, L., Liao, X., Jin, H., & Xue, J. (2026) · IEEE International Symposium on High-Performance Computer Architecture (HPCA) · doi:10.1109/hpca68181.2026.11408598 · Cited by 2
RoMe: Row Granularity Access Memory System for Large Language Models
Nam, H., Baek, S., Kim, J., et al. (2026) · Proceedings - International Symposium on High-Performance Computer Architecture/Proceedings · doi:10.1109/hpca68181.2026.11408542 · Cited by 1
I/O Analysis is All You Need: An I/O Analysis for Long-Sequence Attention
Lu, X., Long, B., Chen, X., Han, Y., & Sun, X. (2026) · International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) · doi:10.1145/3779212.3790174 · Cited by 1
A Cost-Effective Near-Storage Processing Solution for Offline Inference of Long-Context LLMs
Jang, H., Song, J., Shin, C., Noh, S. U., Jung, J., Park, J., & Lee, J. (2026) · International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) · doi:10.1145/3779212.3790119 · Cited by 11
Towards High-Goodput LLM Serving with Prefill-decode Multiplexing
Chen, Y., Cui, W., Zhao, H., Xu, Z. S., Fan, X., Chen, X., Zhou, Y., Sun, S., He, B., & Chen, Q. (2026) · International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) · doi:10.1145/3779212.3790236 · Cited by 23
SpAct-NDP: Efficient LLM Inference via Sparse Activation on NDP-GPU Heterogeneous Architecture
Xu, J., Xie, T., Zhou, Y., Li, J., Lian, Y., Zhu, Z., Wang, Y., & Dai, G. (2026) · Asia and South Pacific Design Automation Conference (ASP-DAC) · doi:10.1109/asp-dac66049.2026.11420624 · Cited by 1
Device-Server Collaborative Speculative Decoding for Real-Time LLM Streaming
Biswas, B. R., Guan, Y., Yin, M., Han, T., & Hou, X. (2025) · Globecom · doi:10.1109/globecom59602.2025.11432593 · Cited by 1
Heterogeneous Computing: The Key to Powering the Future of AI Agent Inference
Zhao, Y., & Liu, J. (2026) · Computer · doi:10.1109/mc.2026.3659288 · Cited by 1
The Cost of Expertise: Understanding MoE Decode Performance
Abuzakuk, S., Balmau, O., Chen, J., et al. (2026) · doi:10.1145/3805621.3807659 · Cited by 1
High-Bandwidth Flash for KV Caches: Endurance and Performance Implications
Kyung, K., Moon, Y. J., Cho, J., et al. (2026) · IEEE Computer Architecture Letters · doi:10.1109/lca.2026.3695938 · Cited by 1
HarMoEny: Efficient Inference of MoE Models
Doucet, Z., Sharma, R. K., de Vos, M., Pires, R. P., Kermarrec, A.-M., & Balmau, O. (2026) · IEEE International Parallel and Distributed Processing Symposium (IPDPS) · doi:10.1109/ipdps65963.2026.00069 · Cited by 1
Strategies and design for increasing AI sustainability
Chien, A. A., Gupta, U., Ren, S., Sriraman, A., & Tomlinson, B. (2026) · Nature Reviews Clean Technology · doi:10.1038/s44359-026-00195-w · Cited by 2
SLO-Aware Disaggregated LLM Serving Over Heterogeneous Edge-Cloud Clusters
Zheng, Y., Wang, S., Su, B., Zhu, A., & Zhu, Z. (2026) · International Conference on Artificial Intelligence, Internet of Things and Cloud Computing Technology (AIoTC) · doi:10.1109/aiotc71250.2026.11689386 · Cited by 2
DistDPU: A Disaggregated DPU Architecture for High-Performance and Cost-Efficient AI Clouds
Mei, H., 宙, 高., Zhu, Y., Wang, L., Yang, P., Pei, C., Chen, C., Li, Z., Zhao, J., Gu, D., et al. (2026) · ACM SIGCOMM Conference (SIGCOMM) · doi:10.1145/3789240.3829161 · Cited by 1
AttentionLib: A Scalable Optimization Framework for Automated Attention Acceleration on FPGA
Liu, Z., Zhou, X., Sun, F., et al. (2025) · doi:10.23919/date64628.2025.10992963 · Cited by 1
Aging-aware CPU Core Management for Embodied Carbon Amortization in Cloud LLM Inference
Hewage, T. B., Ilager, S., Rodriguez, M. A., et al. (2025) · doi:10.1145/3679240.3734608 · Cited by 1
A Component-Centric Perspective on Hardware Accelerators for LLMs
Jia, K., Wang, X., Chen, H., Wei, Z., Xinxiong, L., Zenan, F., & An, F. (2025) · IEEE Access · doi:10.1109/access.2025.3609769 · Cited by 2
KVDirect: Distributed Disaggregated LLM Inference
Chen, S., Jiang, R., Yu, D., Xu, J., Chao, M., Meng, F., Jiang, C., Xu, W., & Liu, H. (2025) · arXiv · doi:10.48550/arxiv.2501.14743 · Cited by 1
“Prefill B Prefill B Figure 2: An example with three LLM inference scheduling approaches according to Splitwise [33].”
InfluentialSmoothAgent: Efficient Long-Horizon LLM-Based Agent Serving with Lookahead Context Engineering
Pan, Z., Wang, Q., Hu, Z., Chen, C., Guan, Y., Zhou, Y., Swanson, S., & Ding, Y. (2026) · arXiv · doi:10.48550/arxiv.2607.00151
“On prefill-only instances [53, 54, 82], lookahead requests consume service time and increase the queuing delay of other requests.”
InfluentialPrefillShare: A Shared Prefill Module for KV Reuse in Multi-LLM Disaggregated Serving
Woo, S., Kim, H., Shim, S., Jo, M., Jeong, H., Lee, J., Kim, J., Lee, S., Ryu, J., Park, B., et al. (2026) · arXiv · doi:10.48550/arxiv.2602.12029
“First, repeated prefill significantly increases latency by both raising time-to-first-token (TTFT) through additional prefill computation and worsening tail latency via intensified contention between prefill and decode in existing serving stacks (Zhong et al., 2024; Patel et al., 2024).”
InfluentialPEEK: Predictive Queue-Informed KV Cache Management for LLM Serving
Xie, B., Wang, Z., Tanaka, M., & Zheng, Z. (2026) · arXiv · doi:10.48550/arxiv.2607.02525
“DistServe [Zhong et al., 2024] and Splitwise [Patel et al., 2024] disaggregate prefill from decode; Mooncake [Qin et al., 2025] extends this to a distributed KV pool; Llumnix [Sun et al., 2024] migrates running requests across instances.”
Unified AI Gateway: A Framework for Joint Model Routing and KV Cache Management
Lu, J., Zhang, X., & Shao, Y. (2026) · arXiv · doi:10.48550/arxiv.2609.06940
“Prefill-decode (PD) dis-aggregation places the compute-intensive prefill phase and the memory-intensive decode phase on different nodes, enabling independent resource management [23], [24].”
CacheRoute: Planned Prefix-Affinity Routing for Large-Scale LLM Serving
Cheng, H. (2026) · arXiv · doi:10.48550/arxiv.2608.19677
“Llumnix migrates requests to repair imbalance [Sun et al.(2024)]; DistServe and Splitwise separate prefill and decode placement [Zhong et al.(2024), Patel et al.(2024)].”
Alviano, M., Capalbo, M., Gottlob, G., Grillo, L., Kareem, I., Lo Scudo, F., Piccolo, S. A., & Rodriguez Reiners, L. A. (2026) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.21678
“Completion tokens are especially critical, as they are generated sequentially and therefore dominate inference time, whereas prompt tokens can be processed more efficiently through parallelization in modern LLM architectures (Patel et al. 2023).”
Towards ML System Extensibility
Deng, W., Ruan, A., Frisella, M., Chen, K.-H., Cho, S., Humphries, J. T., Qiao, R., & Wang, S. (2025) · Workshop on Hot Topics in Operating Systems · doi:10.1145/3713082.3730392
Parallelization Techniques for Large Language Models: A Review from Training to Inference
Liu, S., Tao, X., Cao, W., & Ming, Z. (2025) · Lecture notes in computer science · doi:10.1007/978-981-96-8725-1_25
Design Considerations for LLM Inference in Data Centers: Chip and Interconnect
Shi, Z., Wang, Y., Diao, M., Wu, W., Su, Z., Li, X., Niu, R., Zang, H., Song, K., Yu, Y., et al. (2025) · IEEE Symposium on VLSI Technology and Circuits (VLSI Technology and Circuits) · doi:10.23919/vlsitechnologyandcir65189.2025.11074875
XShift: FPGA-efficient Binarized LLM with Joint Quantization and Sparsification
Zhou, S., Tian, H., Meng, S., Chen, J., Yu, J., & Wang, K. (2025) · ACM/IEEE Design Automation Conference (DAC) · doi:10.1109/dac63849.2025.11133363
Xiong, Y., Zhang, R., Zu, Y., Liu, W., Zhu, Z., Geng, J., Li, B., Cao, Q., & Zhou, X. (2025) · IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems · doi:10.1109/tcad.2025.3624135
Xue, Q., Shi, Y., Wang, B., et al. (2025) · IEEE International Conference on High Performance Computing and Communications (HPCC) · doi:10.1109/hpcc67675.2025.00058
Adaptoserve: An Efficient System for Supporting Adaptive Chunked-Prefills in LLM Inference
Ding, Y., Zhao, J., Cai, Z., Shi, K., Zeng, F., & Yang, B. (2025) · IEEE International Conference on High Performance Computing and Communications; IEEE International Conference on Smart City; IEEE International Conference on Data Science and Systems (HPCC/SmartCity/DSS) · doi:10.1109/hpcc67675.2025.00051
Divide and Compute: Distributed Model Offloading for Computation
Akhilesh, M. K., Bharadwaj, A. R., Prabhu, N. N., et al. (2025) · International Conference on Big Data Analytics (ICBDA) · doi:10.1109/icbda65366.2025.11211291
Xu, X., Yuan, S., Hu, B., et al. (2025) · doi:10.1109/iccad66269.2025.11240806
Improving the Performance of Out-of-Core LLM Inference Using Heterogeneous Host Memory
Gupta, S., & Dwarkadas, S. (2025) · IEEE International Symposium on Workload Characterization · doi:10.1109/iiswc66894.2025.00035
A Performance Study of LLM Inference on Mooncake-Based Clusters
Yoon, J., Ahn, H., Lee, S., Kim, K., Lee, H., Park, J., & Ahn, S. (2025) · IEEE International Conference on Consumer Electronics-Asia (ICCE-Asia) · doi:10.1109/icce-asia67487.2025.11263536
FlexSpark: Robust and Efficient Multi-Device Collaborative Inference over Wireless Network
Shao, Y., Li, H., Hu, S., Du, X., Wu, H., Zhou, J., & Tan, K. (2025) · Asia-Pacific Workshop on Networking (APNet) · doi:10.1145/3735358.3735368
Enabling Cost-Efficient LLM Inference on Mid-Tier GPUs With NMP DIMMs
Kim, H., La, A., & Izraelevitz, J. (2025) · IEEE Computer Architecture Letters · doi:10.1109/lca.2025.3646622
Joint Request Batching and Worker Assignment in AI-RAN Edge Inference Systems
Qiu, N., Gan, S., Wang, X., Feng, C., Chen, X., Han, S., & Wang, X. (2025) · International Conference on Future Communications and Networks (FCN) · doi:10.1109/fcn66513.2025.11296658
Lemix: Unified Scheduling for Llm Training and Inference on Multi-Gpu Systems
Li, Y., Li, Z., Zhu, Y., & Liu, C. (2025) · IEEE Real-Time Systems Symposium (RTSS) · doi:10.1109/rtss66672.2025.00052
C auchy : A Cost-Efficient LLM Serving System through Adaptive Heterogeneous Deployment
Zhang, Y., Shen, H., Yang, R., Tian, D., Luo, Y., Zhang, M., Li, L., Hu, C., Wo, T., Song, C., et al. (2025) · ACM Symposium on Cloud Computing (SoCC) · doi:10.1145/3772052.3772264
Multiplexed Heterogeneous LLM Serving via Stage-Aligned Parallelism
Luo, T., Ng, K. K. W., Khor, Z. P., Sankhe, S., Loo, B. T., & Liu, V. (2025) · ACM Symposium on Cloud Computing (SoCC) · doi:10.1145/3772052.3772207
Zhang, T., Qin, H., Jin, D., Chen, S., He, H., Tan, X., Zhu, S., & Yang, J. (2026) · IEEE Transactions on Cognitive Communications and Networking · doi:10.1109/tccn.2026.3657037
Wang, H., Wang, Y., Yang, J., Wei, S., Hu, Y., & Yin, S. (2025) · IEEE Asian Solid-State Circuits Conference (A-SSCC) · doi:10.1109/a-sscc67472.2025.11349430
PASCAL: A Phase-Aware Scheduling Algorithm for Serving Reasoning-based Large Language Models
Cho, E., Bang, J., Hwang, R., & Rhu, M. (2026) · IEEE International Symposium on High-Performance Computer Architecture (HPCA) · doi:10.1109/hpca68181.2026.11408493 · Cited by 2
LEGO: Supporting LLM-Enhanced Games with One Gaming GPU
Zhao, H., Cui, W., Zhang, Z., Zhang, W., Li, J., Chen, Q., Pang, P., Zijun, L. I., Han, Z., Yang, Y., et al. (2026) · IEEE International Symposium on High-Performance Computer Architecture (HPCA) · doi:10.1109/hpca68181.2026.11408477
Towards Resource-Efficient Serverless LLM Inference with SLINFER
Xu, C., Zijun, L. I., Chen, Q., Zhao, H., Tang, X., & Guo, M. (2026) · IEEE International Symposium on High-Performance Computer Architecture (HPCA) · doi:10.1109/hpca68181.2026.11408548 · Cited by 4
Lim, S., Kim, J.-H., Moon, S., Cha, J., Seo, D., Kim, J., Lee, H., Lee, J., & Kim, J. (2026) · IEEE Journal of Solid-State Circuits · doi:10.1109/jssc.2026.3663603
Reducing LLM Inference Memory Bandwidth via Frequent Exponent Value Encoding
Michalec, M., Tannu, S., & Sohi, G. S. (2026) · IEEE Computer Architecture Letters · doi:10.1109/lca.2026.3671166
APSys: Disaggregated LLM Serving System with an Adaptive Parallel Strategy
Li, Z., Wang, H., Zhang, A., Yan, Z., & Huang, R. (2025) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata66926.2025.11402259
WVA: A Global Optimization Control Plane for llmd
Malvankar, A., Villard, L., Abdi, M., et al. (2026) · IEEE ... International Conference on Cloud Computing · doi:10.1109/cloud72782.2026.00056
BLADE: Boosting LLM Decoding’s Communication Efficiency in DRAM-based PIM
Zhao, Y., Liu, F., Wang, Z., Li, M., Zhang, M., Chen, C., & Jiang, L. (2026) · Asia and South Pacific Design Automation Conference (ASP-DAC) · doi:10.1109/asp-dac66049.2026.11420462
REPA: Re configurable P IM for the Joint A cceleration of KV Cache Offloading and Processing
Hong, Y., Yang, J., Peng, B., & Yao, J. (2026) · International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) · doi:10.1145/3779212.3790212
Next-Generation Sustainable Wireless Systems: Energy Efficiency Meets Environmental Impact
Thomas, C. K., Hashash, O., Ehsani, K., et al. (2025) · Globecom · doi:10.1109/globecom59602.2025.11432202
E$^{2}$LLM: Structure-Guided Efficient Inference for LLMs in Distributed Edge-IoT Environments
Feng, X., Yang, H., Chen, Z., Luo, C., Wang, Y., Zhou, Z., Xu, W., & Leung, V. C. M. (2026) · IEEE Transactions on Mobile Computing · doi:10.1109/tmc.2026.3676689
Game-Based LLM Inference Task Offloading for Edge Computing System
Hou, S., Gan, M., Ni, W., Zhai, Z., & Liu, X. (2026) · IEEE Transactions on Green Communications and Networking · doi:10.1109/tgcn.2026.3676823
Characterizing Performance–Energy Trade-offs of Large Language Models in Multi-Request Workflows
Ifath, M. M. A., & Haque, I. (2026) · Proceedings of the ACM on Measurement and Analysis of Computing Systems · doi:10.1145/3788089 · Cited by 6
Wang, Y., Wu, Z., Chong, Y. J., & Fong, X. (2026) · IEEE Journal on Emerging and Selected Topics in Circuits and Systems · doi:10.1109/jetcas.2026.3679018
EACC: Efficient Agent Context Cache Sharing for Multi-Agent Systems
Cheng, S., Gu, Y., Wu, C., & Meng, J. (2026) · Lecture notes in computer science · doi:10.1007/978-981-95-8405-5_9
Revisiting Disaggregated Large Language Model Serving for Performance and Energy Implications
Li, J., Zhu, Y., Chen, B., Lee, E. K., & Nahrstedt, K. (2026) · Workshop on Machine Learning and Systems (EuroMLSys) · doi:10.1145/3805621.3807662 · Cited by 3
Where the Time Goes: Analysis of a Public LLM Serving System
Demiray, B. K., Yousefzadeh-Asl-Miandoab, E., Garbinato, B., Tözün, P., & Delgado, P. (2026) · Workshop on Machine Learning and Systems (EuroMLSys) · doi:10.1145/3805621.3807650
A Case for a Simulation-Driven Exploration of Distributed GenAI Platforms
Trivedi, A., Stoica, R., Cohn, J., Harnik, D., Zhu, Y., Terner, J., Margalit, G., Schmuck, F., Tarasov, V., & Sundararaman, S. (2026) · Workshop on Machine Learning and Systems (EuroMLSys) · doi:10.1145/3805621.3807623
Towards a Solution to the Management Scaling Paradox in Distributed LLM Inference
Noohi, A., Asoodeh, B., & Barbalace, A. (2026) · doi:10.1145/3805621.3807658
DeCLIP: Pareto-Efficient Collaborative Inference in Decentralized Physical Infrastructure Networks
Du, Y., Leung, C., Liu, J., Wang, Z., & Leung, V. C. M. (2026) · IEEE Open Journal of the Computer Society · doi:10.1109/ojcs.2026.3690486
CALM: Cost-efficient, model Agnostic, Low-latency Modular LLM Deployment framework
Bansal, A., Gill, D. K., Joshi, R., Pham, H., Zamparo, L., & Liu, W. (2026) · International Conference on Advances in Artificial Intelligence and Machine Learning (AAIML) · doi:10.1109/aaiml67890.2026.11498121
Pan, Z., Zou, X., Chen, P., Chen, M., Li, H., & Wang, Y. (2026) · Lecture notes in computer science · doi:10.1007/978-981-92-0378-9_24
Bai, F., Zhang, H., Chen, Z., Du, J., Chen, Z., & Lu, Y. (2026) · Proceedings of the ACM on Management of Data · doi:10.1145/3802009
Efficient LLM Serving for Agentic Workflows: A Data Systems Perspective
Wadlom, N., Shen, J., & Lu, Y. (2026) · Proceedings of the ACM on Management of Data · doi:10.1145/3802046
Hu, Q., & Li, J. (2026) · Electronics · doi:10.3390/electronics15102139
Reddy, G. C. P. (2026) · Research Square · doi:10.21203/rs.3.rs-9776212/v1
Orla: A Library for Serving LLM-Based Multi-Agent Systems
Shahout, R., Tirmazi, H., Yu, M., et al. (2026) · doi:10.1145/3786335.3813227
AdaScheduler: an SLO-aware LLM inference scheduling algorithm without output length predictionPreprint
Liu, L., Zhou, S., Xu, W., et al. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.6826582
HARMONI: Hierarchical ARchitecture MOdeling for LLMs with Near/In Memory Computing
Kiyawat, K., Seneviratne, Y., Fan, Z., et al. (2026) · IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS) · doi:10.1109/ispass69572.2026.00016
Cho, J., Choi, H., Heo, G., & Park, J. (2026) · IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS) · doi:10.1109/ispass69572.2026.00012 · Cited by 1
Optimizing Dropout in LLM Training: Performance Comparison of Fusion and Overlap
Ma, H., Liu, J., & Krashinsky, R. (2026) · Proceedings of the ACM on Measurement and Analysis of Computing Systems · doi:10.1145/3805645
Li, X., Fan, J., Wang, Q., Spatharakis, D., Ghafouri, S., Vandierendonck, H., John, D., Ji, B., Butt, A. R., & Nikolopoulos, D. S. (2026) · Proceedings of the ACM on Measurement and Analysis of Computing Systems · doi:10.1145/3805653 · Cited by 1
ACES: A Chiplet Architecture with Resource Partition and Dynamic Scheduling for Agentic LLMs
Li, H., Li, M., Zhu, Z., & Jia, T. (2026) · Design, Automation, and Test in Europe · doi:10.23919/date69613.2026.11539693
Yu, J., Taneja, A., Lin, J., & Zhang, M. (2026) · ISC High Performance (ISC) · doi:10.23919/isc.2026.11520495
Zhang, B., Hou, J., & Zhang, J. (2026) · IEEE Transactions on Parallel and Distributed Systems · doi:10.1109/tpds.2026.3703757 · Cited by 1
CONCORD: Asynchronous Sparse Aggregation for Device-Cloud Rag Under Document Isolation
Hu, X., Tang, Z., Yao, Z., Wang, T., & Jia, W. (2026) · IEEE International Conference on Web Services (ICWS) · doi:10.1109/icws72778.2026.00134
Chen, X., Guo, F., Liu, C., Peng, M., & Quek, T. Q. S. (2026) · IEEE Wireless Communications and Networking Conference · doi:10.1109/wcnc65185.2026.11555072
Moore, H., Qi, S., Milojicic, D. S., Bash, C. E., & Pasricha, S. (2026) · International Green and Sustainable Computing Conference · doi:10.1145/3797248.3815404
AdaCGen: Heterogeneity-Aware Layer Management for Efficient KV Cache Offloading in LLMs
Wang, Y., Guo, J., Wang, Z., & Tan, Y. (2026) · IEEE International Symposium on Circuits and Systems proceedings · doi:10.1109/iscas66217.2026.11562764
DEMUS: Large Multimodal Model Serving at the Edge with Diffusion-Based Scheduling
Zhang, H., Ma, X., Wang, T., Lin, M., & Li, W. (2026) · IEEE Conference on Computer Communications (INFOCOM) · doi:10.1109/infocom59046.2026.11571478 · Cited by 1
3D-Learning: Diffusion-Augmented Distributionally Robust Decision-Focused Learning
Wen, J., Fan, L., & Yang, J. (2026) · IEEE Conference on Computer Communications (INFOCOM) · doi:10.1109/infocom59046.2026.11571436
Metadata
- Authors
- Patel, P., Choukse, E., Zhang, C., Shah, A., Goiri, Í., Maleki, S., & Bianchini, R.
- Affiliations
- University of Washington (US)
- Microsoft (United States) (US)
- ORCID
- Venue
- 2024 ACM/IEEE 51st Annual International Symposium on Computer Architecture (ISCA)
- Publisher
- IEEE
- Published
- 2024-06
- Pages
- 118-132
- ISSN
- 1063-6897 (ISSN-L)
- OpenAlex
- W4401211704
- Semantic Scholar
- ad9146d98ae95bbeeef460abe083ecc2c4798672
- Cited by
- 298
- References
- 59
- Citations by year
- 2026: 176 · 2025: 108 · 2024: 14
- FWCI
- 90.63
- Type
- conference paper
- Language
- en
- Open access
- closed
- Licence
- Locations
- Proceedings - International Symposium on Computer Architecture/Proceedings (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Hardware and Architecture › Parallel Computing and Optimization Techniques
- Physical Sciences › Computer Science › Information Systems › Cloud Computing and Resource Management
- Physical Sciences › Computer Science › Computer Networks and Communications › Distributed and Parallel Computing Systems
- Keywords
- large language model inference, phase splitting, token generation, GPU clusters, request scheduling, resource management, power efficiency, cost efficiency, throughput, network libraries
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
