The DOI matches its Crossref record; 60 of the 60 references it cites were found in a registry or an index.
Splitwise: Efficient Generative LLM Inference Using Phase Splitting
Abstract
Generative large language model (LLM) applications are growing rapidly, leading to large-scale deployments of expensive and power-hungry GPUs. Our characterization of LLM inference shows that each inference request undergoes two phases: a compute-intensive prompt computation phase and a memory intensive token generation phase, each with distinct latency, throughput, memory, and power characteristics. Despite state-of-the-art batching and scheduling, the token generation phase underutilizes compute resources. Unlike prompt computation, token generation does not need the compute capability of the latest GPUs and can be run with lower power and cost. Based on these insights, we propose Splitwise, a model deployment and scheduling technique that splits the two phases of LLM inference requests on to separate machines. Splitwise enables phase-specific resource management using hardware that is well suited for each phase. Request state is transferred efficiently between machines using optimized network libraries on the fast back-plane interconnects available in today’s GPU clusters. Using Splitwise, we design homogeneous and heterogeneous LLM inference clusters optimized for throughput, cost, and power Compared to current designs, Splitwise clusters achieve up to $1.4 \times$ higher throughput at $\mathbf{2 0 \%}$ lower cost. Alternatively, they can deliver $2.35 \times$ more throughput under the same power and cost budgets.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Llumnix migrates requests to repair imbalance [Sun et al.(2024)]; DistServe and Splitwise separate prefill and decode placement [Zhong et al.(2024), Patel et al.(2024)].”
From CacheRoute: Planned Prefix-Affinity Routing for Large-Scale LLM Serving (2026) · cites this paper as Patel et al.(2024)
In the citing paper
“The second approach is to separate prefill and decoding to different GPUs to avoid prefilldecoding interference Patel et al. (2024); Zhong et al. (2024).”
From Preble: Efficient Distributed Prompt Scheduling for LLM Serving (2024) · cites this paper as Patel et al. (2024)
In the citing paper
“Several recent works have targeted LLM usages with long prompts and proposed solutions to solve the imbalance problem Agrawal et al. (2023; 2024); Zhong et al. (2024); Patel et al. (2024).”
From Preble: Efficient Distributed Prompt Scheduling for LLM Serving (2024) · cites this paper as Patel et al. (2024)
In the citing paper
“increasing aggregate prefill load and worsening tail latency by intensifying prefill-decode interference in existing LLM serving stacks.”
From PrefillShare: A Shared Prefill Module for KV Reuse in Multi-LLM Disaggregated Serving (2026) · cites this paper as Patel et al., 2024
In the citing paper
“Disaggregated serving reduces such interference by placing prefill and decode on separate GPUs”
From PrefillShare: A Shared Prefill Module for KV Reuse in Multi-LLM Disaggregated Serving (2026) · cites this paper as Patel et al., 2024
In the citing paper
“By isolating prefill execution from the decode loop, disaggregated serving prevents prefill computation from interfering with autoregressive decoding, providing more stable ITL in real-world deployments.”
From PrefillShare: A Shared Prefill Module for KV Reuse in Multi-LLM Disaggregated Serving (2026) · cites this paper as Patel et al., 2024
In the citing paper
“On prefill-only instances [53, 54, 82], lookahead requests consume service time and increase the queuing delay of other requests.”
From SmoothAgent: Efficient Long-Horizon LLM-Based Agent Serving with Lookahead Context Engineering (2026) · cites this paper as [53]
In the citing paper
“Prefill-decode disaggregation [53, 82] eliminates interference between prefill and decode requests by dispatching them to separate instances, and has become the de facto design in large-scale datacenter LLM serving.”
From SmoothAgent: Efficient Long-Horizon LLM-Based Agent Serving with Lookahead Context Engineering (2026) · cites this paper as [53]
In the citing paper
“Prefill-decode (PD) disaggregation places the compute-intensive prefill phase and the memory-intensive decode phase on different nodes, enabling independent resource management [23], [24].”
From Unified AI Gateway: A Framework for Joint Model Routing and KV Cache Management (2026) · cites this paper as [23]
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Jakubův, J., Chvalovský, K., Goertzel, Z., et al. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,648
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325 · cited 2×
“Transformer model sizes have grown steadily, from the early BERT models [36] having 340 million parameters, to GPT-3 [28] with a staggering 175 billion parameters, and GPT-4 rumored to have even more.”
BackgroundLanguage Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · cited 3×
“1 Generative LLM inference for a single request consists of several forward passes through the model, since the output tokens are generated one by one.”
BackgroundInfluentialMapReduce: simplified data processing on large clusters
Dean, J., & Ghemawat, S. (2008) · Communications of the ACM · doi:10.1145/1327452.1327492 · Cited by 18,476 · cited 1×
“Distributed dataflow systems orchestrate large-scale computational graphs and aim to provide general-purpose programmability [34], [46], [75], [82].”
BackgroundRoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515
Transformers: State-of-the-Art Natural Language Processing
Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., et al. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations · doi:10.18653/v1/2020.emnlp-demos.6 · Cited by 8,741
The Hadoop Distributed File System
Shvachko, K. V., Kuang, H., Radia, S., & Chansler, R. J. (2010) · Digest of papers - IEEE Symposium on Mass Storage Systems · doi:10.1109/msst.2010.5496972 · Cited by 4,911 · cited 1×
“Distributed dataflow systems orchestrate large-scale computational graphs and aim to provide general-purpose programmability [34], [46], [75], [82].”
BackgroundResilient distributed datasets: a fault-tolerant abstraction for in-memory cluster computing
Zaharia, M., Chowdhury, M., Das, T., Dave, A., Ma, J. H., McCauley, M., Franklin, M. J. M., Shenker, S. J., & Stoica, I. G. (2012) · Cited by 3,622
Research works printed but not linked: 28
- Add Splitwise Implementation to vLLM, GitHub.
- AMD Instinct™ MI250 Accelerator
- Azure InfiniBand HPC VMs
- 2023, Azure Public Dataset: Azure LLM Inference Trace
- CoreWeave - Specialized Cloud Provider
- Google Assistant with Bard
- HPC Interconnect on CoreWeave Cloud
- Intel BigDL-LLM
- Intel Sapphire Rapids with HBM.
- Microsoft Azure ND A100 v4-series
- MSCCL++: A GPU-driven communication stack for scalable AI applications
- Numenta Inference on CPUs
- NVIDIA Accelerated InfiniBand Solutions.
- NVIDIA Chip Shortage
- NVIDIA DGX A100: Universal System for AI Infrastructure.
- NVIDIA DGX H100
- NVIDIA Hopper GPUs Expand Reach as Demand for AI Grows
- OpenAI ChatGPT APIs
- Power Availability Stymies Datacenter Growth
- SplitwiseSim: LLM Serving Cluster Simulator
- The New Bing.
- TurboMind Inference Server
- Patterson, Domain Specific Architectures for Deep Neural Networks: Three Generations of Tensor Processing Units (TPUs), Allen School Distinguished Lecture
- Meta. Introducing the AI Research SuperCluster - Meta’s CuttingEdge AI Supercomputer for AI Research
- Radford, 2019, Language Models are Unsupervised Multitask Learners, OpenAI blog
- Schmid, Fine-tune FLAN-T5 XL/XXL using DeepSpeed & Hugging Face Transformers
- Schmid, Llama 2 is here - Get it on Hugging Face
- Yu, 2022, Orca: A Distributed Serving System for Transformer-Based Generative Models, in OSDI
Cited by
Sorted by citations
Items 201–300 of 309
GenSched: Phase-Aware Generative Scheduling for LLM Inference in Heterogeneous Edge Networks
Jin, H., Li, S., & Gregory, M. A. (2026) · IEEE Conference on Computer Communications (INFOCOM) · doi:10.1109/infocom59046.2026.11571651
Omni-MP: Practical PIM Matrix Mapping for Accelerating LLM Inference on a Real PIM Hardware
Baek, D., Son, J., Choi, J. O., et al. (2026) · doi:10.1145/3774895.3819987
I/O-Aware PIM Acceleration for Long-Sequence LLM Inference with Hybrid Sparse Attention
Lu, X., Hu, L., Huang, H., Jiang, P., & Sun, X. (2026) · IEEE International Parallel and Distributed Processing Symposium (IPDPS) · doi:10.1109/ipdps65963.2026.00046
VoLLM: Smoothness-aware Serving of LLM-powered Voice Q&A via Adaptive Preemption
Wang, X., Li, Y., Ren, J., Pang, L., Yang, S., & Liu, Y. (2026) · IEEE International Parallel and Distributed Processing Symposium (IPDPS) · doi:10.1109/ipdps65963.2026.00037
PEACE: Preemptive and Efficient Cluster Scheduling for LLM Inference with Mixed Prompts
Zhang, Z., & Shen, H. (2026) · IEEE International Parallel and Distributed Processing Symposium (IPDPS) · doi:10.1109/ipdps65963.2026.00105
Padalkar, K., Jesus, V., & Calero, J. M. A. (2026) · Workshop on Advanced Tools, Programming Languages, and PLatforms for Implementing and Evaluating Algorithms for Distributed Systems (ApPLIED) · doi:10.1145/3820355.3820384
Aurora: A Disaggregated GPU-PNM-PIM System for High-Throughput Mixed-Length LLM Inference
Kim, H., Yu, S., Kim, M., Lee, J., Sung, H., & Lee, E. (2026) · International Conference on Supercomputing · doi:10.1145/3797905.3800536
Not All Errors Are Equal: A Systematic Study of Error Propagation in Large Language Model Inference
Huang, Y., Di, S., & Li, G. (2026) · International Conference on Supercomputing · doi:10.1145/3797905.3800513 · Cited by 2
SparseServe: Unlocking Parallelism for Dynamic Sparse Attention in Long-Context LLM Serving
Zhou, Q., Yin, P., Zuo, P., Wang, C., & Cheng, J. S.-C. (2026) · International Conference on Supercomputing · doi:10.1145/3797905.3800528 · Cited by 12
Latency-SLO-Aware Memory Offloading for Large Language Model Inference
Ma, C., Zhao, H., Ye, Z., Yang, Z., Fu, T., Han, J., Zhang, J., Luo, Y., Wang, X., Wang, Z., et al. (2026) · International Conference on Supercomputing · doi:10.1145/3797905.3807874 · Cited by 3
Phase-aware Peak Power Reduction for Minimizing the Capital Expense of LLM Inference
Wu, S., Ma, Y., & Wang, X. (2026) · International Conference on Supercomputing · doi:10.1145/3797905.3815788
ET: An Energy Efficient Edge Transformer Architecture
Nag, S., Bacellar, A. T. L., Jha, A., John, E. B., Kailas, K. K., Lima, P. M. V., Yadwadkar, N. J., França, F. M. G., & John, L. K. (2026) · ACM Transactions on Architecture and Code Optimization · doi:10.1145/3828726
Liu, Y., Lan, W., Li, J., Yuan, B., & Yang, H. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-37422-6_34
Overcoming Congestion in Edge MoE: A Bi-Level Game-Theoretic Orchestration Framework
Chen, Y. (2026) · International Conference on Signal Processing, Communication and Control Systems (SPCCS) · doi:10.1109/spccs69331.2026.11588633
LLMSGHD: Co-Designing Large Language Model Software and Hardware for Efficient Inference
Xue, Q., Ma, S., Xu, R., Shi, Y., Wang, B., Luo, S., Zhang, X., Huang, L., & Li, T. (2026) · ACM Transactions on Architecture and Code Optimization · doi:10.1145/3820045
Omnia: Efficient RAG Serving through Speculative Scheduling
Fu, R., Li, S., Xu, Y., Wu, T., Ma, Z., & Shi, J. (2026) · ACM International Symposium on High-Performance Parallel and Distributed Computing (HPDC) · doi:10.1145/3806645.3807814
Dynamo-MoE: Accelerating Sparse Large Model Inference with Dynamic Parallelization
Chen, J., Li, S., Fu, R., Wu, T., Ma, Z., & Dong, J. K. (2026) · ACM International Symposium on High-Performance Parallel and Distributed Computing (HPDC) · doi:10.1145/3806645.3807820
Gullapalli, S., & Divi, V. R. (2026) · International Conference on Electronics Representation and Algorithm (ICERA) · doi:10.1109/icera72709.2026.11666706
Elastic Memory Remapping for Multi-tenant LLM Serving
Li, R., Pal, S., Pullu, V. N., Sinha, P., Ryoo, J., John, L. K., & Yadwadkar, N. J. (2026) · ACM SIGOPS Operating Systems Review · doi:10.1145/3830422.3830427
Breaking Memory Wall for Fast Edge LLM Inference Using Contextual Sparsity
Wei, Z., Zhou, Y., Zhao, J., Li, K., & Fu, X. (2026) · IEEE Transactions on Mobile Computing · doi:10.1109/tmc.2026.3712257
Heterogeneous Low-Power Agentic Architecture for Edge Distributed Home Surveillance
Zappia, L. L., Lattuada, M., Bianchessi, M. A., & Pau, D. P. (2026) · Electronics · doi:10.3390/electronics15143175 · Cited by 1
Wang, H., Wang, Y., Yang, J., Hu, Y., & Yin, S. (2026) · IEEE Journal of Solid-State Circuits · doi:10.1109/jssc.2026.3708458
Phase-Wise Analysis of LLM Inference Acceleration on GPU, CPU, and Edge Device
Das, S., Cheng, J., Vallabhajosyula, M. S., Soni, B., & Ramnath, R. (2026) · Practice and Experience in Advanced Research Computing · doi:10.1145/3785462.3815902
A Privacy-preserving Edge-cloud Video Analytics System via Policy-based Frame Transformation
Lu, R., Wu, W., Shi, S., et al. (2026) · ACM Transactions on Internet Technology · doi:10.1145/3833875
I/o for LLM inference: a survey of storage and memory bottlenecks
Chowdhury, R. (2026) · Artificial Intelligence Review · doi:10.1007/s10462-026-11651-1
Unified Static-Dynamic Pruning for Efficient LLM Inference
Kim, J., Lee, Y., & Do, J. (2026) · Proceedings of the VLDB Endowment · doi:10.14778/3836663.3836665 · Cited by 1
Dynamic Task Scheduling and Function Orchestration for Serverless LLM
Kellari, M., Diamanti, C., Spatharakis, D., et al. (2026) · IEEE International Conference on High Performance Switching and Routing (HPSR) · doi:10.1109/hpsr68369.2026.11615182
LLMEdger: Phase-Aware Model Parallelism Scheduler for LLM Inference on Edge
Shen, X., & Mashayekhy, L. (2026) · IEEE International Conference on Fog and Edge Computing (ICFEC) · doi:10.1109/icfec69006.2026.00010
Anwar, M. S., Ravi, A., Ghosh, I., et al. (2026) · IEEE International Symposium on a World of Wireless, Mobile and Multimedia Networks (WoWMoM) · doi:10.1109/wowmom69805.2026.00029
LLM-Pilot: SLO-Aware and Cost-Efficient LLM Serving on Public Cloud VM Clusters via Offloading
Kim, J., Kim, K., Chung, H., Yang, J., Kim, J. J., Li, D., & Kim, Y. (2026) · IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (CCGrid) · doi:10.1109/ccgrid68966.2026.00023
Towards Multi-Model LLM Schedulers: Empirical Insights Into Offloading and Preemption
Yildiz, M., Spadaccino, P., Rolich, A., Cuomo, F., & Baiocchi, A. (2026) · Mediterranean Artificial Intelligence and Networking Conference (MAIN) · doi:10.1109/main71116.2026.11622343
Huang, Z., Jia, W., Li, Y., Lin, X., Fan, S., Tang, C., Deng, S., & Liu, Y. (2026) · Proceedings - International Symposium on Computer Architecture/Proceedings · doi:10.1109/isca66397.2026.00054
Tetris: Efficient Long-context LLM Serving with Chunkwise Dynamic Sequence Parallelism
Li, C., Yang, Y., Zheng, X., Yang, Q., Guan, Y., Zheng, S., Chang, L., Liu, S., Liu, X., & Sun, G. (2026) · Proceedings - International Symposium on Computer Architecture/Proceedings · doi:10.1109/isca66397.2026.00098
EVA: Accelerating LLM Decoding via an Efficient Vector Quantization Architecture
Duan, B., Guo, C., Wei, C., et al. (2026) · ACM/IEEE International Symposium on Computer Architecture (ISCA) · doi:10.1109/isca66397.2026.00102
PipeWeave: Synergizing Analytical and Learning Models for Unified GPU Performance Prediction
Zhang, K., Cui, Y., Zhang, S., Ding, C., Qian, S., Wang, L., Cao, J., Xue, G., Huang, C., Yang, G., et al. (2026) · Proceedings - International Symposium on Computer Architecture/Proceedings · doi:10.1109/isca66397.2026.00126
He, L., Zhou, F., Peng, C., Dong, S., Zhang, Z., Yang, H., Liu, Y., Zhang, G., & Jia, H. (2026) · Proceedings - International Symposium on Computer Architecture/Proceedings · doi:10.1109/isca66397.2026.00058
Accelerator Polymorphism: Transcending Domain-Specific Architectures with Robotics
Xu, H., Oh, S., Lee, Y., Rajan, A. R. A., Mahapatra, R., Patil, O., Li, Y., Park, J., & Esmaeilzadeh, H. (2026) · Proceedings - International Symposium on Computer Architecture/Proceedings · doi:10.1109/isca66397.2026.00065 · Cited by 1
Cassandra: Enabling Reasoning LLMs at Edge via Self-Speculative Decoding
Choi, S., Kim, Y., Son, M., & Kim, J.-Y. (2026) · Proceedings - International Symposium on Computer Architecture/Proceedings · doi:10.1109/isca66397.2026.00022 · Cited by 1
Optimizing AI: A Reference Architecture to Streamline AI Inference and Performance Acceleration
Jovanovic, M., Campbell, M., & Campbell, M. (2026) · Computer · doi:10.1109/mc.2026.3695984
DynoPipe: Heterogeneous Edge-Cloud LLM Serving with Dynamically Orchestrated Pipeline Boundaries
Lin, Y., Chen, B., Zhang, X., Xu, C., & Ye, K. (2026) · Proceedings - International Symposium on Computer Architecture/Proceedings · doi:10.1109/isca66397.2026.00077
FlashServe: Adaptive Kernel Provision for Quantized LLM Serving
Chen, Y., Wang, H., Liang, J., et al. (2026) · IEEE International Conference on Distributed Computing Systems (ICDCS) · doi:10.1109/2575-8411.2026.00078
ConServe: Contiguity-Preserving Memory Management for Multi-Turn LLM Serving
Li, B. (2026) · Proceedings - International Symposium on Computer Architecture/Proceedings · doi:10.1109/isca66397.2026.00075
Tang, Z., Gong, W., Liwang, M., Kang, M., & Li, L. (2026) · International Conference on Control and Automation · doi:10.1109/icca69928.2026.11618065
Fleet-Level Optimization for Heterogeneous Device Provisioning in ML Model Deployments
Jiang, H. H., & Joe‐Wong, C. (2026) · IEEE International Conference on Distributed Computing Systems Workshops (ICDCSW) · doi:10.1109/icdcsw72724.2026.00038
ShadowLLM: Resource-Efficient Hot Standby for Heterogeneous Edge LLM Serving
Chen, Z., Ding, P., Zhu, Y., Zhang, H., Lv, J., Huang, Y., & Qiao, X. (2026) · IEEE International Conference on Distributed Computing Systems (ICDCS) · doi:10.1109/2575-8411.2026.00051
Symbiotic MLLM Serving: Dynamically Balancing Parallelism Across GPUs and Resources Within GPUs
Li, Z., Zhao, J., Zhang, Y., Duan, Z., Liu, X., Li, S., Zhang, S., Li, S., Yu, D., Wen, Y., et al. (2026) · Proceedings - International Symposium on Computer Architecture/Proceedings · doi:10.1109/isca66397.2026.00173
Rearchitecting the Datacenter Lifecycle for AI
Stojkovic, J., Zhang, C., Goiri, Í., & Bianchini, R. (2026) · Proceedings - International Symposium on Computer Architecture/Proceedings · doi:10.1109/isca66397.2026.00029 · Cited by 1
Self-citationOrionInfer: Low-Overhead Parallelism Switching and Live Migration for Efficient LLM Serving
Feng, J., Yang, G., Huang, Y., Liang, S., Huang, C., Yan, M., & Wu, J. (2026) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3770855.3817626
KVServe: Service-Aware KV Cache Compression for Communication-Efficient Disaggregated LLM Serving
Liu, Z., Ma, X., Luo, D., Zhao, H., Lu, B., Huang, W., Gu, Y., Liu, X., Wei, Z., Liu, J., et al. (2026) · Proceedings of the ACM SIGCOMM 2026 Conference · doi:10.1145/3789240.3829139
ARK: Avoiding Routing Collisions for KV Cache Transfer in Disaggregated LLM Inference
Lin, H.-C., Hsu, T.-W., Ho, C.-E., & Saeed, A. (2026) · ACM SIGCOMM Conference (SIGCOMM) · doi:10.1145/3789240.3828750
MindFusion: Communication-Efficient Mobile-Cloud Orchestration for MLLMs
Zeng, X., & Huang, Q. (2026) · IEEE International Workshop on Foundation Models for Cyber-Physical Systems & Internet of Things (FMSys) · doi:10.1109/fmsys72110.2026.00012
Optimization of the Generation Phase in Post-Training of Large Language Models Based on RLHF
Zhang, C., Wan, M., Wang, J., Li, K., Wang, Y., & Cao, R. (2026) · IEEE International Conference on Big Data Security on Cloud, IEEE International Conference on High Performance and Smart Computing, and IEEE International Conference on Intelligent Data and Security (BigDataSecurity/HPSC/IDS) · doi:10.1109/hpsc69479.2026.00019
Self-citationJiang, F., Zhang, J., & Li, X. (2026) · Communications in computer and information science · doi:10.1007/978-981-92-0288-1_26
Lyapunov-Guided KV Cache Control for Multi-tenant Edge LLM Servicing
Yu, H., Seah, W. K. G., Xu, G., & Xu, X. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-35251-4_13
Performance Characterization and Optimization of LLM Inference on Tenstorrent AI Accelerators
Lim, J., Shin, D., Kim, U., Choi, J., Kim, K. M., Shin, S., Park, S., Koo, G., & Suh, T. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-35251-4_32
Asynchronous Speculative Sentence Decoding with Step-Level Alignment Classification
Hu, X., Li, Y., Ye, K., & Gao, X. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-36493-7_9
Gupta, R. (2026) · IEEE International Parallel and Distributed Processing Symposium Workshops (IPDPSW) · doi:10.1109/ipdpsw71298.2026.00091
Congestion-Aware Scheduling for Heterogeneous LLM-Agent Teams
Xie, W., & Li, Y. (2026) · IEEE Annual International Computer Software and Applications Conference (COMPSAC) · doi:10.1109/compsac69091.2026.00432
Oracle: A BERT-Based Workload Predictor for LLM Inference
Zhao, B., Sun, X., Zhou, H., Shi, W., & Huang, X. (2026) · International Conference on Optical Communications and Networks (ICOCN) · doi:10.1109/icocn70692.2026.11656052
TokenSimV2: Accurate and Fast LLM Inference on GPUs via Dynamic Graph Modeling
Bian, Z., Wu, F., Li, Z., Zhao, Y., Wu, J., Yang, X., & Zhuo, Y. (2026) · Lecture notes in computer science · doi:10.1007/978-981-92-4805-6_9
WaferSim: A Simulation Infrastructure for LLM Service on Wafer-Scale Chips
Xu, Z. M., Liu, J., Kong, D., et al. (2026) · Lecture notes in computer science · doi:10.1007/978-981-92-4805-6_39
UNAS: Urgency- and Fairness-Aware Scheduling for SLO-Oriented LLM Serving
Fu, T., Liao, J., Chen, X., et al. (2026) · IEEE International Conference on Joint Cloud Computing (JCC) · doi:10.1109/jcc72984.2026.00058
ParaCascade: A Parallel Cascading Framework Supporting Early Routing
Wei, H., Yin, L., Chen, C., Sun, J., Zhang, M., & Peng, Y. (2026) · IEEE International Conference on Joint Cloud Computing (JCC) · doi:10.1109/jcc72984.2026.00031
Pegasus: Accelerating Large Language Model Inference with Stateful Prefix Caching
Chen, F., Li, P., Yu, D., & Guo, S. (2026) · IEEE International Conference on Joint Cloud Computing (JCC) · doi:10.1109/jcc72984.2026.00017
TAILOR: Token-Aware Partitioning and Routing for Edge–Cloud Transformer Inference
Huang, X., Liu, R. R., & Wu, J. (2026) · International Workshop on Quality of Service · doi:10.1109/iwqos70441.2026.11661040
UPServe: Backend Agnostic Proxy for Black-box Heterogeneous LLM Scheduling
Wan, H., Hei, C., Li, F., Gao, C., Jia, Y., Liu, T., & Wang, X. (2026) · International Workshop on Quality of Service · doi:10.1109/iwqos70441.2026.11661114
BYSTANDER: State-Aware End-to-End Latency Prediction for Heterogeneous LLM Inference Scheduling
Jung, M., Kim, H., Park, H., Lee, H., & Yu, H. (2026) · IEEE ... International Conference on Cloud Computing · doi:10.1109/cloud72782.2026.00037
FlowGuard: Slack-Aware Overload Control for Multi-Agent LLM Serving
Sadiq, A. Z., & Shen, H. (2026) · International Conference on Computer Communications and Networks (ICCCN) · doi:10.1109/icccn69946.2026.11662939
WaSMa: Workflow-Aware Scheduling for Multi-Agent LLM Systems
Rao, U., Sadiq, A. Z., Shen, H., Yang, Y., Hodges, E.-A. R., & Muntean, M. (2026) · International Conference on Computer Communications and Networks (ICCCN) · doi:10.1109/icccn69946.2026.11662938
Managing KV Cache for Coordinated Waiting and Execution Time in LLM Serving
Shen, H., Sen, T., & Tanaka, M. (2026) · International Conference on Computer Communications and Networks (ICCCN) · doi:10.1109/icccn69946.2026.11662646
M-CSN: Joint Architecture and Flow Scheduling for Metro-Scale AI Fabric Based on Supernodes
Guo, L., Zhao, J., Quan, W., Wang, S., Li, J., & Zhang, H. (2026) · IEEE Transactions on Network Science and Engineering · doi:10.1109/tnse.2026.3727990
UTARI: An On-Prem GPU-Accelerated LLM Platform for Higher-Education Services
Rif’an, M., Amastini, F., Rudiana, D., Wicaksono, A., Rinusantoro, S., & Adompo, A. R. (2026) · doi:10.1109/icoict71118.2026.11664754
To Keep or Not to Keep: Learning KV Cache Retention in Disaggregated LLM Serving Systems
Liu, D., Yu, Y., Jiang, E., Wang, S., & Wu, Y. N. (2026) · ACM International Systems and Storage Conference (SYSTOR) · doi:10.1145/3793230.3837769
Nguyen, C. T., Hoang, D. T., Nguyen, D. N., et al. (2026) · IEEE Transactions on Network and Service Management · doi:10.1109/tnsm.2026.3730014
QueueBreak: A Trace-to-Diagnosis Pipeline for Tail Latency in Agentic LLM Services
Ma, X., Li, Z., Dong, C., Shao, D., & Liu, X. (2026) · IEEE International Conference on Web Services (ICWS) · doi:10.1109/icws72778.2026.00129
QoS-Aware Prefix Orchestration for Geo-Distributed LLM-as-a-Service
Ge, Z., & Chen, L. (2026) · IEEE International Conference on Web Services (ICWS) · doi:10.1109/icws72778.2026.00179
EcoFlow: Cost-Aware LLM Serving on Heterogeneous GPUs via Min-Cost Max-Flow
Jiang, F., Kou, F., Zhang, J., et al. (2026) · IEEE International Conference on Web Services (ICWS) · doi:10.1109/icws72778.2026.00182
Dong, D., Wu, C., Chen, Z., Yan, L., Niu, J., Yu, J., Ma, Z., & Zhai, J. (2026) · IEEE Transactions on Parallel and Distributed Systems · doi:10.1109/tpds.2026.3732699
Multi-Stage LLM Scheduling Based on Privacy and Latency in Edge-Cloud Collaboration
Meng, B., Xu, X., Zou, Y., Liao, Y., Chen, X., & Xiao, M. (2026) · International Joint Conference on Information and Communication Engineering (JCICE) · doi:10.1109/jcice71302.2026.11682296
PreServe: Intelligent Management for LMaaS Systems via Hierarchical Prediction
Jiang, Z., Huang, Y., Yu, G., Huang, J., Gu, J., & Lyu, M. R.-T. (2026) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3744916.3773173
RPSC: Robust LLM Scheduling by Tolerating Prediction Inaccuracy and Mitigating Tail Latency
Xiao, G., Mo, X., Li, J., & Wu, W. (2026) · Proceedings of the International Conference on Parallel Processing · doi:10.1145/3832810.3832840
Heterogeneous SLO Guaranteed Multi-Resource-Aware Batching in LLM Serving
Shen, H., Sen, T., & He, Y. (2026) · Proceedings of the International Conference on Parallel Processing · doi:10.1145/3832810.3832922
WAQ-LLM: Optimizing Multi-Instance LLM Deployment via Workload-Aware Queueing Model
Lai, J., Luo, Y., & Wang, Q. (2026) · Proceedings of the International Conference on Parallel Processing · doi:10.1145/3832810.3832865
CrossServe: Cross-Layer Scheduling for SLO Optimization in Multi-Tenant LLM Serving
Li, J., & Li, J. (2026) · Proceedings of the International Conference on Parallel Processing · doi:10.1145/3832810.3832859
Context-Aware LLM-Guided Neighbourhood Search for Simulation-Based Production Scheduling
Skapinyecz, R. (2026) · Machine Learning and Knowledge Extraction · doi:10.3390/make8090284
Unlocking Software-defined GPU Fabric Scheduling in the LLM Era
Chen, D., Gu, Y., Huang, Y., et al. (2026) · ACM SIGOPS Asia-Pacific Workshop on Systems (APSys) · doi:10.1145/3838177.3841731
Disaggregated LLM KV-Cache Storage Requires Coordination-Free Consistency Abstractions
Pullela, K., & Acharya, H. B. (2026) · ACM Workshop on Hot Topics in Storage and File Systems (HotStorage) · doi:10.1145/3837053.3837339
The Storage Wall in LEO Compute Clouds
Bhosale, V., Gavrilovska, A., & Bhardwaj, K. (2026) · ACM Workshop on Hot Topics in Storage and File Systems (HotStorage) · doi:10.1145/3837053.3837355
LLM KV-cache: To Restore or To Recompute, That Is the Question
Najafizadeh, A., Tarasov, V., Merenstein, A., Zhu, Y. F., & Zadok, E. (2026) · ACM Workshop on Hot Topics in Storage and File Systems (HotStorage) · doi:10.1145/3837053.3837336
Periodicity-Aware AI Data Center IT Load Forecasting Across Heterogeneous Cluster Scenarios
Zong, M., Xu, J., Ke, D., Jiang, X., Jiang, W., & Yuan, Y. (2026) · IEEE/IAS Industrial and Commercial Power System Asia (I&CPS Asia) · doi:10.1109/icpsasia70813.2026.11692219
AI Data-Center Flexibility is Conditional, Not Intrinsic
Yang, Y., Chen, Y., Yuan, M., et al. (2026) · Energy Use · doi:10.59717/ipj.energy-use.2026.100074
PRESAGE: Uncertainty-Aware Predictive Elastic Scheduling for Cloud Analytics and GenAI Inference
Wang, Y., Zou, X., Zhou, L., Wei, K., & Rhee, M. (2026) · IEEE Access · doi:10.1109/access.2026.3734498 · Cited by 3
Learning to Compress: Energy-Aware Large Language Model Serving via Adaptive KV Caching
Cheng, Y., Shao, Z., Yu, N., Fu, Y., & Dong, Y. (2026) · IEEE Power Engineering Society General Meeting · doi:10.1109/pesgm58988.2026.11693290
MeshRT: Compile-Time Governed Wafer-Scale Runtime for Low-Latency High-Throughput Inference
He, C., Xu, L., Lu, Z., Huang, Y., Xiao, H., Deng, C., Ma, L., Miao, Z., Yang, F., & Mai, L. (2026) · ACM SIGOPS Symposium on Operating Systems Principles (SOSP) · doi:10.1145/3830418.3843856
Buoy: Efficient and Effective Cache Replacement for Prefix Caching
Wang, L., Jia, R., Wang, K., Lü, K., & Wan, J. (2026) · Proceedings of the ACM on Management of Data · doi:10.1145/3837101
Janus: Multi-LLM Serving at Production Scale
Zhou, T., Wang, Y., Zhou, Y., Liu, Z., Wang, Z., Peng, Y., Wang, Y., Zhang, Y., Yin, J., Tian, K., et al. (2026) · ACM SIGOPS Symposium on Operating Systems Principles (SOSP) · doi:10.1145/3830418.3843867
StreamEP: Straggler-Tolerant MoE Decoding without Communication Barriers
Liang, Y., Wang, S., Song, J., Zhou, Y., Kim, G.-W., He, G., & Park, S. J. (2026) · ACM SIGOPS Symposium on Operating Systems Principles (SOSP) · doi:10.1145/3830418.3843869
SNART: Real-Time Translation via ECAPA-TDNN-Based Biometric Speaker Filtering
Szabó, G. (2026) · IEEE Conference on Information Technology and Data Science (CITDS) · doi:10.1109/citds70571.2026.11706857
Носко, С. В., Hulevych, M., Коломійцев, О., Третяк, В. Ф., Галицький, О. Ф., Гайбадулов, Б. В., Моргун, Є. В., Клєвцов, С., Корольков, К., Бєлоус, М., et al. (2026) · InterConf · doi:10.51582/interconf.19-20.10.2026.026
Metadata
- Authors
- Patel, P., Choukse, E., Zhang, C., Shah, A., Goiri, Í., Maleki, S., & Bianchini, R.
- Affiliations
- University of Washington (US)
- Microsoft (United States) (US)
- ORCID
- Venue
- 2024 ACM/IEEE 51st Annual International Symposium on Computer Architecture (ISCA)
- Publisher
- IEEE
- Published
- 2024-06
- Pages
- 118-132
- ISSN
- 1063-6897 (ISSN-L)
- OpenAlex
- W4401211704
- Semantic Scholar
- ad9146d98ae95bbeeef460abe083ecc2c4798672
- Cited by
- 298
- References
- 59
- Citations by year
- 2026: 176 · 2025: 108 · 2024: 14
- FWCI
- 90.63
- Type
- conference paper
- Language
- en
- Open access
- closed
- Licence
- Locations
- Proceedings - International Symposium on Computer Architecture/Proceedings (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Hardware and Architecture › Parallel Computing and Optimization Techniques
- Physical Sciences › Computer Science › Information Systems › Cloud Computing and Resource Management
- Physical Sciences › Computer Science › Computer Networks and Communications › Distributed and Parallel Computing Systems
- Keywords
- large language model inference, phase splitting, token generation, GPU clusters, request scheduling, resource management, power efficiency, cost efficiency, throughput, network libraries
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
