Splitwise: Efficient Generative LLM Inference Using Phase Splitting
Abstract
Generative large language model (LLM) applications are growing rapidly, leading to large-scale deployments of expensive and power-hungry GPUs. Our characterization of LLM inference shows that each inference request undergoes two phases: a compute-intensive prompt computation phase and a memory intensive token generation phase, each with distinct latency, throughput, memory, and power characteristics. Despite state-of-the-art batching and scheduling, the token generation phase underutilizes compute resources. Unlike prompt computation, token generation does not need the compute capability of the latest GPUs and can be run with lower power and cost. Based on these insights, we propose Splitwise, a model deployment and scheduling technique that splits the two phases of LLM inference requests on to separate machines. Splitwise enables phase-specific resource management using hardware that is well suited for each phase. Request state is transferred efficiently between machines using optimized network libraries on the fast back-plane interconnects available in today’s GPU clusters. Using Splitwise, we design homogeneous and heterogeneous LLM inference clusters optimized for throughput, cost, and power Compared to current designs, Splitwise clusters achieve up to $1.4 \times$ higher throughput at $\mathbf{2 0 \%}$ lower cost. Alternatively, they can deliver $2.35 \times$ more throughput under the same power and cost budgets.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Llumnix migrates requests to repair imbalance; DistServe and Splitwise separate prefill and decode placement.
“Llumnix migrates requests to repair imbalance [Sun et al.(2024)]; DistServe and Splitwise separate prefill and decode placement [Zhong et al.(2024), Patel et al.(2024)].”
From CacheRoute: Planned Prefix-Affinity Routing for Large-Scale LLM Serving (2026) · cites this paper as Patel et al.(2024)
Separating prefill and decoding to different GPUs avoids prefill-decoding interference.
“The second approach is to separate prefill and decoding to different GPUs to avoid prefilldecoding interference Patel et al. (2024); Zhong et al. (2024).”
From Preble: Efficient Distributed Prompt Scheduling for LLM Serving (2024) · cites this paper as Patel et al. (2024)
Several recent works have targeted LLM usages with long prompts and proposed solutions to solve the imbalance problem.
“Several recent works have targeted LLM usages with long prompts and proposed solutions to solve the imbalance problem Agrawal et al. (2023; 2024); Zhong et al. (2024); Patel et al. (2024).”
From Preble: Efficient Distributed Prompt Scheduling for LLM Serving (2024) · cites this paper as Patel et al. (2024)
Redundant prefill computation and duplicated KV caches increase aggregate prefill load and worsen tail latency by intensifying prefill-decode interference in existing LLM serving stacks.
“increasing aggregate prefill load and worsening tail latency by intensifying prefill-decode interference in existing LLM serving stacks.”
From PrefillShare: A Shared Prefill Module for KV Reuse in Multi-LLM Disaggregated Serving (2026) · cites this paper as Patel et al., 2024
Disaggregated serving reduces prefill-decode interference by placing prefill and decode on separate GPUs.
“Disaggregated serving reduces such interference by placing prefill and decode on separate GPUs”
From PrefillShare: A Shared Prefill Module for KV Reuse in Multi-LLM Disaggregated Serving (2026) · cites this paper as Patel et al., 2024
Disaggregated serving prevents prefill computation from interfering with autoregressive decoding, providing more stable ITL.
“By isolating prefill execution from the decode loop, disaggregated serving prevents prefill computation from interfering with autoregressive decoding, providing more stable ITL in real-world deployments.”
From PrefillShare: A Shared Prefill Module for KV Reuse in Multi-LLM Disaggregated Serving (2026) · cites this paper as Patel et al., 2024
On prefill-only instances, lookahead requests consume service time and increase the queuing delay of other requests.
“On prefill-only instances [53, 54, 82], lookahead requests consume service time and increase the queuing delay of other requests.”
From SmoothAgent: Efficient Long-Horizon LLM-Based Agent Serving with Lookahead Context Engineering (2026) · cites this paper as [53]
Prefill-decode disaggregation eliminates interference between prefill and decode requests by dispatching them to separate instances, and has become the de facto design in large-scale datacenter LLM serving.
“Prefill-decode disaggregation [53, 82] eliminates interference between prefill and decode requests by dispatching them to separate instances, and has become the de facto design in large-scale datacenter LLM serving.”
From SmoothAgent: Efficient Long-Horizon LLM-Based Agent Serving with Lookahead Context Engineering (2026) · cites this paper as [53]
Prefill-decode (PD) disaggregation places the compute-intensive prefill phase and the memory-intensive decode phase on different nodes, enabling independent resource management.
“Prefill-decode (PD) disaggregation places the compute-intensive prefill phase and the memory-intensive decode phase on different nodes, enabling independent resource management [23], [24].”
From Unified AI Gateway: A Framework for Joint Model Routing and KV Cache Management (2026) · cites this paper as [23]
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · see citing works
Jakubův, J., Chvalovský, K., Goertzel, Z., et al. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,648 · see citing works
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 48,293 · cited 2× · see citing works
“Transformer model sizes have grown steadily, from the early BERT models [36] having 340 million parameters, to GPT-3 [28] with a staggering 175 billion parameters, and GPT-4 rumored to have even more.”
BackgroundLanguage Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · cited 3× · see citing works
“1 Generative LLM inference for a single request consists of several forward passes through the model, since the output tokens are generated one by one.”
BackgroundInfluentialMapReduce: simplified data processing on large clusters
Dean, J., & Ghemawat, S. (2008) · Communications of the ACM · doi:10.1145/1327452.1327492 · Cited by 18,476 · cited 1× · see citing works
“Distributed dataflow systems orchestrate large-scale computational graphs and aim to provide general-purpose programmability [34], [46], [75], [82].”
BackgroundRoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515 · see citing works
Kaugeranna, R., Kaugeranna, E., & Aion, (. S. 4. 6. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.giscience.2023.43 · Cited by 13,695 · see citing works
Transformers: State-of-the-Art Natural Language Processing
Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., et al. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations · doi:10.18653/v1/2020.emnlp-demos.6 · Cited by 8,741 · see citing works
The Hadoop Distributed File System
Shvachko, K. V., Kuang, H., Radia, S., & Chansler, R. J. (2010) · Digest of papers - IEEE Symposium on Mass Storage Systems · doi:10.1109/msst.2010.5496972 · Cited by 4,911 · cited 1× · see citing works
“Distributed dataflow systems orchestrate large-scale computational graphs and aim to provide general-purpose programmability [34], [46], [75], [82].”
BackgroundResilient distributed datasets: a fault-tolerant abstraction for in-memory cluster computing
Zaharia, M., Chowdhury, M., Das, T., Dave, A., Ma, J. H., McCauley, M., Franklin, M. J. M., Shenker, S. J., & Stoica, I. G. (2012) · Cited by 3,622 · see citing works
Isard, M., Budiu, M., Yu, Y., Birrell, A. D., & Fetterly, D. (2007) · European Conference on Computer Systems · doi:10.1145/1272996.1273005 · Cited by 2,433 · cited 1× · see citing works
“Distributed dataflow systems orchestrate large-scale computational graphs and aim to provide general-purpose programmability [34], [46], [75], [82].”
Efficient Memory Management for Large Language Model Serving with PagedAttention
Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez, J., Zhang, H., & Stoica, I. (2023) · Proceedings of the 29th Symposium on Operating Systems Principles · doi:10.1145/3600006.3613165 · Cited by 1,817 · cited 5× · see citing works
“However, once the request is in the token phase, we count it as one active token, since the tokens are generated one at a time (assuming a beam search size of one [51]).”
BackgroundMethodInfluentialEfficient Memory Management for Large Language Model Serving with PagedAttention
Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez, J. E., Zhang, H., & Stoica, I. (2023) · arXiv · doi:10.48550/arxiv.2309.06180 · Cited by 1,802 · see citing works
FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness
Dao, T., Fu, D. Y., Ermon, S., Rudra, A., & Ré, C. (2022) · arXiv · doi:10.48550/arxiv.2205.14135 · Cited by 1,214 · cited 1× · see citing works
“LLM inference serving is a rapidly developing field, with several recent works optimizing batching [23], [25], [51], [53], [81], scheduling [22], [42], [51], [66], [73], [79], and memory usage [32], [35], [51], [74].”
MethodRay: A Distributed Framework for Emerging AI ApplicationsPreprint
Moritz, P., Nishihara, R., Wang, S., Tumanov, A., Liaw, R., Liang, E., Elibol, M., Yang, Z., Paul, W., Jordan, M. I., et al. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1712.05889 · Cited by 953 · cited 1× · see citing works
“LLM inference under Splitwise can be viewed as a static computational graph with two stages, so it could be implemented using distributed frameworks that provide efficient GPU abstractions [59].”
MethodSchwarzkopf, M., Konwinski, A., Abd-El-Malek, M., & Wilkes, J. (2013) · European Conference on Computer Systems · doi:10.1145/2465351.2465386 · Cited by 651 · cited 1× · see citing works
“Insights from prior work on partitioned or replicated scheduling could help improve scalability [27], [61], [72] and are orthogonal to Splitwise.”
BackgroundOusterhout, K., Wendell, P., Zaharia, M., & Stoica, I. G. (2013) · ACM SIGOPS Symposium on Operating Systems Principles (SOSP) · doi:10.1145/2517349.2522716 · Cited by 586 · cited 1× · see citing works
“Insights from prior work on partitioned or replicated scheduling could help improve scalability [27], [61], [72] and are orthogonal to Splitwise.”
BackgroundOpt: Open pre-trained transformer language models
Zhang, S. (2025) · TIB Data Manager · doi:10.57702/dn0pmcbs · Cited by 405 · see citing works
Single-ISA heterogeneous multi-core architectures: the potential for processor power reduction
Kumar, R., Farkas, K. I., Jouppi, N. P., Ranganathan, P., & Tullsen, D. M. (2003) · IEEE/ACM International Symposium on Microarchitecture (MICRO) · doi:10.1109/micro.2003.1253185 · Cited by 373 · cited 1× · see citing works
“Research on heterogeneous multiprocessor CPU scheduling attempts to match workload heterogeneity to hardware heterogeneity [29], [40], [41], [50], [76], [80].”
BackgroundApollo: scalable and coordinated scheduling for cloud-scale computing
Boutin, É., Ekanayake, J., Lin, W., Shi, B., Zhou, J., Qian, Z., Wu, M., & Zhou, L. (2014) · Operating Systems Design and Implementation · doi:10.5555/2685048.2685071 · Cited by 324 · see citing works
LLM.int8(): 8-Bit Matrix Multiplication for Transformers at Scale
Dettmers, T., Lewis, M., Belkada, Y., & Zettlemoyer, L. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-2198 · Cited by 305 · see citing works
DeepSpeed- Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scale
Aminabadi, R. Y., Rajbhandari, S., Awan, A. A., Li, C., Li, D., Zheng, E., Ruwase, O., Smith, S., Zhang, M., Rasley, J., et al. (2022) · SC22: International Conference for High Performance Computing, Networking, Storage and Analysis · doi:10.1109/sc41404.2022.00051 · Cited by 297 · cited 1× · see citing works
“LLM inference serving is a rapidly developing field, with several recent works optimizing batching [23], [25], [51], [53], [81], scheduling [22], [42], [51], [66], [73], [79], and memory usage [32], [35], [51], [74].”
MethodRevisiting network support for RDMA
Mittal, R., Shpiner, A., Panda, A., et al. (2018) · doi:10.1145/3230543.3230557 · Cited by 263 · cited 1× · see citing works
“The alternative could be HPC clouds, with Infiniband connections through the CPU [3], or Ethernet, using RoCE [58].”
MethodFast Distributed Inference Serving for Large Language ModelsPreprint
Wu, B., Zhong, Y.-M., Zhang, Z.-L., Liu, S.-Y., Liu, F.-Y., Sun, Y.-H., Huang, G., Liu, X.-Z., & Jin, X. (2023) · arXiv.org · doi:10.48550/arxiv.2305.05920 · Cited by 248 · cited 1× · see citing works
“LLM inference serving is a rapidly developing field, with several recent works optimizing batching [23], [25], [51], [53], [81], scheduling [22], [42], [51], [66], [73], [79], and memory usage [32], [35], [51], [74].”
MethodKwon, Y., Lee, Y., & Rhu, M. (2019) · IEEE/ACM International Symposium on Microarchitecture (MICRO) · doi:10.1145/3352460.3358284 · Cited by 233 · cited 1× · see citing works
“Prior work exploits such heterogeneity to selectively schedule requests between CPUs and accelerators [38], [52], colocate models with complementary memory usage [30], and partition compute/memory on heterogeneous hardware resources [45], [48].”
BackgroundAnalysis of join-the-shortest-queue routing for web server farms
Gupta, V., Harchol‐Balter, M., Sigman, K., et al. (2007) · Performance Evaluation · doi:10.1016/j.peva.2007.06.012 · Cited by 219 · cited 1× · see citing works
“CLS uses Join the Shortest Queue (JSQ) scheduling [39], [85] to assign a prompt and a token machine to each request.”
MethodDeepRecSys: A System for Optimizing End-To-End At-Scale Neural Recommendation Inference
Gupta, U., Hsia, S., Saraph, V., Wang, X., Reagen, B., Wei, G.-Y., Lee, H.-H. S., Brooks, D. M., & Wu, C.-J. (2020) · Proceedings - International Symposium on Computer Architecture/Proceedings · doi:10.1109/isca45697.2020.00084 · Cited by 176 · cited 2× · see citing works
“Prior work exploits such heterogeneity to selectively schedule requests between CPUs and accelerators [38], [52], colocate models with complementary memory usage [30], and partition compute/memory on heterogeneous hardware resources [45], [48].”
BackgroundMArk: Exploiting Cloud Services for Cost-Effective, SLO-Aware Machine Learning Inference Serving.
Zhang, C., Yu, M., Wang, W., & Yan, F. (2019) · Rare & Special e-Zone (The Hong Kong University of Science and Technology) · Cited by 172 · see citing works
Self-citationKannan, R. S., Subramanian, L., Raju, A., Ahn, J., Mars, J., & Tang, L. (2019) · European Conference on Computer Systems · doi:10.1145/3302424.3303958 · Cited by 152 · cited 1× · see citing works
“Prior work on video and ML serving focuses on scheduling model chains with data dependencies under latency constraints [24], [31], [43], [49], [68].”
BackgroundScheduling heterogeneous multi-cores through performance impact estimation (PIE)
Craeynest, K. V., Jaleel, A., Eeckhout, L., et al. (2012) · Proceedings - International Symposium on Computer Architecture/Proceedings · doi:10.1109/isca.2012.6237019 · Cited by 136 · see citing works
Efficient program scheduling for heterogeneous multi-core processors
Chen, J., & John, L. K. (2009) · Proceedings - ACM IEEE Design Automation Conference · doi:10.1145/1629911.1630149 · Cited by 127 · cited 1× · see citing works
“Research on heterogeneous multiprocessor CPU scheduling attempts to match workload heterogeneity to hardware heterogeneity [29], [40], [41], [50], [76], [80].”
BackgroundServing DNNs like Clockwork: Performance Predictability from the Bottom UpPreprint
Gujarati, A., Karimi, R. J., Alzayat, S., Hao, W., Kaufmann, A., Vigfússon, Ý., & Mace, J. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.02464 · Cited by 118 · cited 1× · see citing works
“Since LLM requests are much longer than typical ML requests [37], [38], they incur lower scheduling overhead for similar cluster sizes.”
BackgroundEfficiently Scaling Transformer Inference
Pope, R., Douglas, S., Chowdhery, A., Devlin, J., Bradbury, J., Levskaya, A., Heek, J., Xiao, K., Agrawal, S., & Dean, J. (2022) · arXiv · doi:10.48550/arxiv.2211.05102 · Cited by 113 · cited 1× · see citing works
“LLM inference serving is a rapidly developing field, with several recent works optimizing batching [23], [25], [51], [53], [81], scheduling [22], [42], [51], [66], [73], [79], and memory usage [32], [35], [51], [74].”
MethodCrankshaw, D., Sela, G.-E., Mo, X., Zumar, C., Stoica, I. G., Gonzalez, J. E., & Tumanov, A. (2020) · ACM Symposium on Cloud Computing (SoCC) · doi:10.1145/3419111.3421285 · Cited by 105 · cited 1× · see citing works
“Prior work on video and ML serving focuses on scheduling model chains with data dependencies under latency constraints [24], [31], [43], [49], [68].”
BackgroundCentaur: A Chiplet-based, Hybrid Sparse-Dense Accelerator for Personalized Recommendations
Hwang, R., Kim, T., Kwon, Y., & Rhu, M. (2020) · Proceedings - International Symposium on Computer Architecture/Proceedings · doi:10.1109/isca45697.2020.00083 · Cited by 104 · cited 1× · see citing works
“Prior work exploits such heterogeneity to selectively schedule requests between CPUs and accelerators [38], [52], colocate models with complementary memory usage [30], and partition compute/memory on heterogeneous hardware resources [45], [48].”
BackgroundINFaaS: Automated Model-less Inference Serving
Romero, F., Li, Q., Yadwadkar, N. J., & Kozyrakis, C. (2021) · USENIX Annual Technical Conference · Cited by 102 · see citing works
Characterizing Power Management Opportunities for LLMs in the Cloud
Patel, P., Choukse, E., Zhang, C., Goiri, Í., Warrier, B., Mahalingam, N., & Bianchini, R. (2024) · International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) · doi:10.1145/3620666.3651329 · Cited by 93 · cited 3× · see citing works
“Finally, power optimization is attractive for a CSP, since it enables more GPUs to be deployed in the same datacenter [62], [63], but it may not be as important to the user.”
BackgroundMethodSelf-citationFlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPU
Sheng, Y., Zheng, L., Yuan, B., Li, Z., Ryabinin, M., Fu, D. Y., Xie, Z., Chen, B., Barrett, C., Gonzalez, J. E., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.06865 · Cited by 86 · cited 1× · see citing works
“LLM inference serving is a rapidly developing field, with several recent works optimizing batching [23], [25], [51], [53], [81], scheduling [22], [42], [51], [66], [73], [79], and memory usage [32], [35], [51], [74].”
MethodSelf-citationThe Datacenter as a Computer: Designing Warehouse-Scale Machines, Third Edition
Barroso, L. A., Hölzle, U., & Ranganathan, P. (2018) · Synthesis lectures on computer architecture · doi:10.2200/s00874ed3v01y201809cac046 · Cited by 76 · see citing works
Exploiting heterogeneity for tail latency and energy efficiency
Haque, M. E., He, Y., Elnikety, S., Nguyen, T. D., Bianchini, R., & McKinley, K. S. (2017) · IEEE/ACM International Symposium on Microarchitecture (MICRO) · doi:10.1145/3123939.3123956 · Cited by 55 · cited 1× · see citing works
“Research on heterogeneous multiprocessor CPU scheduling attempts to match workload heterogeneity to hardware heterogeneity [29], [40], [41], [50], [76], [80].”
BackgroundSelf-citationDeja Vu: Contextual Sparsity for Efficient LLMs at Inference TimePreprint
Liu, Z., Wang, J., Dao, T., Zhou, T., Yuan, B., Song, Z., Shrivastava, A., Zhang, C., Tian, Y., Ré, C., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2310.17157 · Cited by 51 · cited 1× · see citing works
“To further reduce our bandwidth utilization, we could also compress the KV-cache before transferring it across the network [55].”
MethodSelf-citationAlpaServe: Statistical Multiplexing with Model Parallelism for Deep Learning ServingPreprint
Li, Z., Zheng, L., Zhong, Y., Liu, V., Sheng, Y., Jin, X., Huang, Y., Chen, Z., Zhang, H., Gonzalez, J. E., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2302.11665 · Cited by 43 · see citing works
FleetRec: Large-Scale Recommendation Inference on Hybrid GPU-FPGA Clusters
Jiang, W., He, Z., Zhang, S., et al. (2021) · doi:10.1145/3447548.3467139 · Cited by 36 · cited 1× · see citing works
“Prior work exploits such heterogeneity to selectively schedule requests between CPUs and accelerators [38], [52], colocate models with complementary memory usage [30], and partition compute/memory on heterogeneous hardware resources [45], [48].”
BackgroundSelf-citationFlashDecoding++: Faster Large Language Model Inference on GPUsPreprint
Hong, K., Dai, G., Xu, J., Mao, Q., Li, X., Liu, J., Chen, K., Dong, Y., & Wang, Y. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2311.01282 · Cited by 35 · cited 1× · see citing works
“LLM inference serving is a rapidly developing field, with several recent works optimizing batching [23], [25], [51], [53], [81], scheduling [22], [42], [51], [66], [73], [79], and memory usage [32], [35], [51], [74].”
MethodHu, Y., Ghosh, R., & Govindan, R. (2021) · ACM Symposium on Cloud Computing (SoCC) · doi:10.1145/3472883.3486993 · Cited by 33 · cited 1× · see citing works
“Prior work on video and ML serving focuses on scheduling model chains with data dependencies under latency constraints [24], [31], [43], [49], [68].”
BackgroundHaque, M. E., Eom, Y. H., He, Y., et al. (2015) · ACM SIGPLAN Notices · doi:10.1145/2775054.2694384 · Cited by 29 · see citing works
Self-citationTowards Improved Power Management in Cloud GPUs
Patel, P., Gong, Z., Rizvi, S., et al. (2023) · IEEE Computer Architecture Letters · doi:10.1109/lca.2023.3278652 · Cited by 27 · cited 1× · see citing works
“This is especially important when building GPU clusters, since GPUs consume much higher power than regular compute machines [63], [64].”
BackgroundSelf-citationSchedTune: A Heterogeneity-Aware GPU Scheduler for Deep Learning
Albahar, H., Dongare, S., Du, Y., et al. (2022) · IEEE International Symposium on Cluster, Cloud and Internet Computing (CCGrid) · doi:10.1109/ccgrid54584.2022.00079 · Cited by 24 · cited 1× · see citing works
“Prior work on video and ML serving focuses on scheduling model chains with data dependencies under latency constraints [24], [31], [43], [49], [68].”
BackgroundSARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefills
Agrawal, A., Panwar, A., Mohan, J., Kwatra, N., Gulavani, B. S., & Ramjee, R. (2023) · arXiv · doi:10.48550/arxiv.2308.16369 · Cited by 20 · cited 3× · see citing works
“LLM inference serving is a rapidly developing field, with several recent works optimizing batching [23], [25], [51], [53], [81], scheduling [22], [42], [51], [66], [73], [79], and memory usage [32], [35], [51], [74].”
BackgroundMethodFairness in Serving Large Language ModelsPreprint
Sheng, Y., Cao, S., Li, D., Zhu, B., Li, Z., Zhuo, D., Gonzalez, J. E., & Stoica, I. G. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2401.00588 · Cited by 10 · cited 1× · see citing works
“LLM inference serving is a rapidly developing field, with several recent works optimizing batching [23], [25], [51], [53], [81], scheduling [22], [42], [51], [66], [73], [79], and memory usage [32], [35], [51], [74].”
MethodPOLCA: Power Oversubscription in LLM Cloud ProvidersPreprint
Patel, P., Choukse, E., Zhang, C., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2308.12908 · Cited by 8 · cited 1× · see citing works
“Finally, power optimization is attractive for a CSP, since it enables more GPUs to be deployed in the same datacenter [62], [63], but it may not be as important to the user.”
MethodSelf-citationAnalysis of JSQ policy on soft real-time scheduling in cluster
Zhu, W. (2000) · doi:10.1109/hpc.2000.846562 · Cited by 2 · cited 1× · see citing works
“Layer-wise transfer also enables other optimizations, such as earlier start of the token phase in the token machines, as well as earlier release of KV-cache memory on the prompt machines.”
MethodHera: A Heterogeneity-Aware Multi-Tenant Inference Server for Personalized Recommendations
Choi, Y., Kim, J., & Rhu, M. (2025) · International Conference on Parallel Architectures and Compilation Techniques (PACT) · doi:10.1109/pact65351.2025.00037 · Cited by 2 · cited 1× · see citing works
“Prior work exploits such heterogeneity to selectively schedule requests between CPUs and accelerators [38], [52], colocate models with complementary memory usage [30], and partition compute/memory on heterogeneous hardware resources [45], [48].”
BackgroundHybrid Computing for Interactive Datacenter ApplicationsPreprint
Patel, P., Lim, K., Jhunjhunwalla, K., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2304.04488 · Cited by 1 · cited 1× · see citing works
“Prior work has studied heterogeneous scheduling for a variety of interactive services [65], [68], [83].”
BackgroundSelf-citationBERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805
OPT: Open Pre-trained Transformer Language Models
Zhang, S., Roller, S., Goyal, N., Artetxe, M., Chen, M., Chen, S., Dewan, C., Diab, M., Li, X., Lin, X. V., et al. (2022) · arXiv · doi:10.48550/arxiv.2205.01068
BLOOM: A 176B-Parameter Open-Access Multilingual Language Model
Workshop, B., :, Scao, T. L., Fan, A., Akiki, C., Pavlick, E., Ilić, S., Hesslow, D., Castagné, R., Luccioni, A. S., et al. (2022) · arXiv · doi:10.48550/arxiv.2211.05100
Flagged first, then by citations
2 of the 59 references couldn’t be matched to a known paper.
Research works printed but not linked: 29
- Add Splitwise Implementation to vLLM, GitHub.
- AMD Instinct™ MI250 Accelerator
- Azure InfiniBand HPC VMs
- 2023, Azure Public Dataset: Azure LLM Inference Trace
- CoreWeave - Specialized Cloud Provider
- Google Assistant with Bard
- HPC Interconnect on CoreWeave Cloud
- Intel BigDL-LLM
- Intel Sapphire Rapids with HBM.
- Microsoft Azure ND A100 v4-series
- MSCCL++: A GPU-driven communication stack for scalable AI applications
- Numenta Inference on CPUs
- NVIDIA Accelerated InfiniBand Solutions.
- NVIDIA Chip Shortage
- NVIDIA DGX A100: Universal System for AI Infrastructure.
- NVIDIA DGX H100
- NVIDIA Hopper GPUs Expand Reach as Demand for AI Grows
- OpenAI ChatGPT APIs
- Power Availability Stymies Datacenter Growth
- SplitwiseSim: LLM Serving Cluster Simulator
- The New Bing.
- TurboMind Inference Server
- Patterson, Domain Specific Architectures for Deep Neural Networks: Three Generations of Tensor Processing Units (TPUs), Allen School Distinguished Lecture
- Meta. Introducing the AI Research SuperCluster - Meta’s CuttingEdge AI Supercomputer for AI Research
- Radford, 2019, Language Models are Unsupervised Multitask Learners, OpenAI blog
- Schmid, Fine-tune FLAN-T5 XL/XXL using DeepSpeed & Hugging Face Transformers
- Schmid, Llama 2 is here - Get it on Hugging Face
- Yang, 2017, PowerChief Intelligent Power Allocation for Multi-stage Applications to Improve Responsiveness on Power Constrained CMP, in ISCA
- Yu, 2022, Orca: A Distributed Serving System for Transformer-Based Generative Models, in OSDI
Cited by
Sorted by citations
EdgeShard: Efficient LLM Inference via Collaborative Edge Computing
Zhang, M., Shen, X., Cao, J., Cui, Z., & Jiang, S. (2024) · IEEE Internet of Things Journal · doi:10.1109/jiot.2024.3524255 · Cited by 253 · see citing works
“This work is distinguished from existing work by partitioning the LLMs and allocating to multiple GPUs in cloud data centers, such as Gpipe [17], PipeDream [18], and Splitwise [19].”
DynamoLLM: Designing LLM Inference Clusters for Performance and Energy Efficiency
Stojkovic, J., Zhang, C., Goiri, Í., Torrellas, J., & Choukse, E. (2025) · 2025 IEEE International Symposium on High Performance Computer Architecture (HPCA) · doi:10.1109/hpca61900.2025.00102 · Cited by 112 · see citing works
“To meet these ever increasing computing demands, researchers have proposed various software [9], [27], [37], [75], [85] and hardware [4], [51], [81] techniques that improve the performance of LLM inference clusters.”
InfluentialSelf-citationTowards Efficient Generative Large Language Model Serving: A Survey from Algorithms to Systems
Miao, X., Oliaro, G., Zhang, Z., Cheng, X., Jin, H., Chen, T., & Jia, Z. (2025) · ACM Computing Surveys · doi:10.1145/3754448 · Cited by 107 · see citing works
“Among the related works, [164] comes closest in subject matter exploring the design of more general Transformer models and domain-specific accelerators.”
Yigit, Y., Ferrag, M. A., Ghanem, M. C., Sarker, I. H., Maglaras, L. A., Chrysoulas, C., Moradpoor, N., Tihanyi, N., & Janicke, H. (2025) · Sensors · doi:10.3390/s25061666 · Cited by 69 · see citing works
Reconciling the contrasting narratives on the environmental impact of large language models
Ren, S., Tomlinson, B., Black, R. W., & Torrance, A. W. (2024) · Scientific Reports · doi:10.1038/s41598-024-76682-6 · Cited by 68 · see citing works
“As a result, the total computational demand of LLM inference can exceed that of training by far 26 , making the amortized training cost for each LLM inference request small in real systems.”
BackgroundLLMCompass: Enabling Efficient Hardware Design for Large Language Model Inference
Zhang, H., Ning, A., Prabhakar, R. B., & Wentzlaff, D. (2024) · Proceedings - International Symposium on Computer Architecture/Proceedings · doi:10.1109/isca59077.2024.00082 · Cited by 60 · see citing works
“Figure 13 indicates our proposed latency-oriented design and throughput-oriented design are around the sweet point.”
BurstGPT: A Real-World Workload Dataset to Optimize LLM Serving Systems
Wang, Y., Chen, Y., Li, Z., Kang, X., Fang, Y., Zhou, Y., Zheng, Y., Tang, Z., He, X. J., Guo, R., et al. (2025) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3711896.3737413 · Cited by 53 · see citing works
Luo, H., Liu, Y., Zhang, R., Wang, J., Sun, G., Niyato, D. T., Yu, H., Xiong, Z., Wang, X., & Shen, X. (2025) · IEEE Transactions on Cognitive Communications and Networking · doi:10.1109/tccn.2025.3612760 · Cited by 52 · see citing works
“Techniques like split inference, that is, partitioning the model layers between device and cloud, are also explored to cope with this demand, albeit introducing complexity in return [76].”
Accelerating Retrieval-Augmented Generation
Quinn, D., Nouri, M., Patel, N., Salihu, J., Salemi, A., Lee, S., Zamani, H., & Alian, M. (2025) · Proceedings of the 30th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 1 · doi:10.1145/3669940.3707264 · Cited by 48 · see citing works
“Nevertheless, the current state-of-the-art focus in building AI systems is only on accelerating the generation phase [5, 8, 21, 38, 52, 57, 66, 84, 101, 102, 105].”
vAttention: Dynamic Memory Management for Serving LLMs without PagedAttention
Prabhu, R., Nayak, A., Mohan, J., Ramjee, R., & Panwar, A. (2025) · International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) · doi:10.1145/3669940.3707256 · Cited by 42 · see citing works
“This is called the KV cache [36, 63, 78] which accounts for a majority of GPU memory usage during inference.”
InfluentialLLM Inference Serving: Survey of Recent Advances and Opportunities
Li, B., Jiang, Y., Gadepally, V. N., & Tiwari, D. (2024) · IEEE High Performance Extreme Computing Conference · doi:10.1109/hpec62836.2024.10938426 · Cited by 42 · see citing works
“Splitwise [31] extensively characterizes the differences in the execution and utilization patterns of the prefill and decoding stage on different generations of GPUs (heterogeneous hardware).”
Yun, S., Kyung, K., Cho, J., Choi, J., Kim, J., Kim, B., Lee, S., Sohn, K., & Ahn, J. H. (2024) · IEEE/ACM International Symposium on Microarchitecture (MICRO) · doi:10.1109/micro61859.2024.00105 · Cited by 40 · see citing works
“TBT and T2FT are critical performance metrics for LLM inference, especially in conversational tasks [43], which often involve multiple rounds of dialogues between the user and the chatbot.”
InfluentialAchieving Peak Performance for Large Language Models: A Systematic Review
Rostam, Z. R. K., Szénási, S., & Kertész, G. (2024) · IEEE Access · doi:10.1109/access.2024.3424945 · Cited by 39 · see citing works
“LightSeq [63] is a lightweight inference library, addresses the need for efficient and convenient deployment of Trans-former models in online services.”
BackgroundMethodInfluentialCE-CoLLM: Efficient and Adaptive Large Language Models Through Cloud-Edge Collaboration
Jin, H., & Wu, Y. (2025) · IEEE International Conference on Web Services (ICWS) · doi:10.1109/icws67624.2025.00046 · Cited by 39 · see citing works
Helix: Serving Large Language Models over Heterogeneous GPUs and Network via Max-Flow
Mei, Y., Zhuang, Y., Miao, X., Yang, J., Jia, Z., & Vinayak, R. (2025) · Proceedings of the 30th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 1 · doi:10.1145/3669940.3707215 · Cited by 35 · see citing works
“6.3 evaluates the fidelity of the simulator and shows that the simulation errors are less than 5% for all metrics.”
InfluentialTAPAS: Thermal- and Power-Aware Scheduling for LLM Inference in Cloud Platforms
Stojkovic, J., Zhang, C., Goiri, Í., Choukse, E., Qiu, H., Fonseca, R., Torrellas, J., & Bianchini, R. (2025) · International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) · doi:10.1145/3676641.3716025 · Cited by 44 · see citing works
“TAPAS enhances cooling and power oversubscription capabilities while effectively managing infrastructure failures, thereby reducing datacenter TCO with minimal impact on workload performance or result quality.”
InfluentialSelf-citationLi, C., Yin, Y., Wu, X., Zhu, J., Gao, Z., Niu, D., Wu, Q., Si, X., Xie, Y., Zhang, C., et al. (2025) · Proceedings - International Symposium on Computer Architecture/Proceedings · doi:10.1145/3695053.3731008 · Cited by 40 · see citing works
“There are two methods for this problem: (1) Duplicate weights for both prefill and decoding stages like prefill-decoding disaggregation [27, 28, 61, 64, 90].”
Self-citationA Survey: Collaborative Hardware and Software Design in the Era of Large Language Models
Guo, C., Cheng, F., Du, Z., Kiessling, J., Ku, J., Li, S., Li, Z., Ma, M., Molom-Ochir, T., Morris, B., et al. (2025) · IEEE Circuits and Systems Magazine · doi:10.1109/mcas.2024.3476008 · Cited by 32 · see citing works
“SpotServe [88] addresses the challenges of using cloud-based preemptible GPU resources by implementing strategies for dynamic adjustment and state recovery, ensuring robust service Authorized licensed use limited to the terms of the applicable license agreement with IEEE.”
SOFA: A Compute-Memory Optimized Sparsity Accelerator via Cross-Stage Coordinated Tiling
Wang, H., Fang, J., Tang, X., Yue, Z., Li, J., Qin, Y., Guan, S., Yang, Q., Wang, Y., Li, C., et al. (2024) · IEEE/ACM International Symposium on Microarchitecture (MICRO) · doi:10.1109/micro61859.2024.00093 · Cited by 34 · see citing works
“Large language models (LLMs) have driven the trans-former architecture to unprecedented levels of complexity”
Leveraging Large Language Models for Enhancing Safety in Maritime Operations
Miller, T. I., Durlik, I., Kostecka, E., Łobodzińska, A., Łazuga, K., & Kozlovska, P. (2025) · Applied Sciences · doi:10.3390/app15031666 · Cited by 31 · see citing works
Low-Overhead General-Purpose Near-Data Processing in CXL Memory Expanders
Ham, H., Hong, J., Park, G., Shin, Y., Woo, O., Yang, W., Bae, J., Park, E., Sung, H., Lim, E., et al. (2024) · IEEE/ACM International Symposium on Microarchitecture (MICRO) · doi:10.1109/micro61859.2024.00051 · Cited by 29 · see citing works
“Thus, we offload it with NDP, using the output vector of SLS as µ thread pool region.”
Generative AI as a Service in 6G Edge-Cloud: Generation Task Offloading by In-Context Learning
Zhou, H., Hu, C., Yuan, D., Yuan, Y., Wu, D., Liu, X., Han, Z., & Zhang, J. (2024) · IEEE Wireless Communications Letters · doi:10.1109/lwc.2024.3520995 · Cited by 29 · see citing works
“We assume a small-scale LLM is deployed at the network edge as in constraint (4c), in which t ,i , t TTFT,edge , and t TPOT,edge represent total generation time, TTFT, and TPOT values.”
POD-A <scp>ttention</scp> : Unlocking Full Prefill-Decode Overlap for Faster LLM Inference
Kamath, A. K., Prabhu, R., Mohan, J., Peter, S., Ramjee, R., & Panwar, A. (2025) · Proceedings of the 30th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2 · doi:10.1145/3676641.3715996 · Cited by 26 · see citing works
“Large-scale service providers often depend on expensive high-end GPUs to meet peak demand or latency targets [41].”
InfluentialYao, Z., Tang, Z., Yang, W., & Jia, W. (2025) · IEEE Transactions on Services Computing · doi:10.1109/tsc.2025.3562362 · Cited by 28 · see citing works
“Patel et al. [37] distribute computation-heavy prompt calculations and memory-heavy token generation across various servers to enhance throughput.”
Sustainable LLM Serving: Environmental Implications, Challenges, and Opportunities : Invited Paper
Ding, Y., & Shi, T. (2024) · International Green and Sustainable Computing Conference · doi:10.1109/igsc64514.2024.00016 · Cited by 24 · see citing works
“LLM serving involves two phases: prefill and decoding [9].”
SpInfer: Leveraging Low-Level Sparsity for Efficient Large Language Model Inference on GPUs
Fan, R., Yu, X., Dong, P., Li, Z., Gong, G., Wang, Q., Wang, W., & Chu, X. (2025) · European Conference on Computer Systems · doi:10.1145/3689031.3717481 · Cited by 24 · see citing works
“Additionally, as inference systems increasingly adopt a de-coupled prefill and decode phase architecture [63, 67, 68, 101], SpInfer’s optimization for the decode phase makes it well-suited for scalable deployment.”
InstAttention: In-Storage Attention Offloading for Cost-Effective Long-Context LLM Inference
Pan, X., Li, E., Li, Q., Liang, S., Shan, Y., Zhou, K., Luo, Y., Wang, X., & Zhang, J. (2025) · IEEE International Symposium on High-Performance Computer Architecture (HPCA) · doi:10.1109/hpca61900.2025.00113 · Cited by 38 · see citing works
“Other solutions [19], [33], [52], [53], [80] leverage disaggregated resources (i.e., GPU, CPU and memory pools) to store the KV cache and accelerate long-context LLM inference, which are not suitable for resource-constrained scenarios.”
ShuffleInfer: Disaggregate LLM Inference for Mixed Downstream Workloads
Hu, C.-C., Huang, H., Xu, L., Chen, X., Wang, C., Xu, J., Chen, S., Feng, H., Wang, S., Bao, Y., et al. (2025) · ACM Transactions on Architecture and Code Optimization · doi:10.1145/3732941 · Cited by 73 · see citing works
“(a) Prefill Schedule: Splitwise [41], Distserve [62], and Dejavu [49] use request batching strategies.”
InfluentialKim, H., Wang, N., Xia, Q., Huang, J., Yazdanbakhsh, A., & Kim, N. S. (2025) · Proceedings - International Symposium on Computer Architecture/Proceedings · doi:10.1145/3695053.3731092 · Cited by 21 · see citing works
Luo, X., Liu, D., Kong, H., Huai, S., Chen, H., Xiong, G., & Liu, W. (2024) · ACM Transactions on Embedded Computing Systems · doi:10.1145/3701728 · Cited by 24 · see citing works
throttLL’eM: Predictive GPU Throttling for Energy Efficient LLM Inference Serving
Kakolyris, A. K., Masouros, D., Vavaroutsos, P., Xydis, S., & Soudris, D. (2025) · IEEE International Symposium on High-Performance Computer Architecture (HPCA) · doi:10.1109/hpca61900.2025.00103 · Cited by 20 · see citing works
“The E2E SLO is set to the p 99 th response time under maximum load.”
InfluentialWindServe: Efficient Phase-Disaggregated LLM Serving with Stream-based Dynamic Scheduling
Feng, J., Huang, Y., Zhang, R., Liang, S., Yan, M., & Wu, J. (2025) · Proceedings of the 52nd Annual International Symposium on Computer Architecture · doi:10.1145/3695053.3730999 · Cited by 16 · see citing works
“In the domain of LLM serving systems, there is a significant body of advancing work focused on enhancing the efficiency and performance of large language model inference [7, 18, 24, 26, 28, 29, 31, 33, 34, 42, 45].”
InfluentialQuality-of-Service Aware LLM Routing for Edge Computing With Multiple Experts
Yang, J., Wu, Q., Feng, Z., Zhou, Z., Guo, D., & Chen, X. (2025) · IEEE Transactions on Mobile Computing · doi:10.1109/tmc.2025.3590969 · Cited by 16 · see citing works
“Some other systems focus on GPU kernel optimization and kernel fusion [21], model parallelism [22], [23], batching algorithm [11], [18], [19], KV-cache management [12], [24] and disaggregated inference [25], [26].”
Performance Characterization of CXL Memory and Its Use Cases
Wang, X., Liu, J., Wu, J., Yang, S., Ren, J., Shankar, B., & Li, D. (2025) · IEEE International Parallel and Distributed Processing Symposium (IPDPS) · doi:10.1109/ipdps64566.2025.00097 · Cited by 16 · see citing works
“LLM can be memory-consuming and have execution phases sensitive to memory band-width [27], [46], [54], hence having potentials to benefit from CXL.”
Liao, J., Chang, F., Xue, Y., Xia, T., Huang, Z., & Wang, Y. (2026) · International Conference on Communications, Information System and Computer Engineering (CISCE) · doi:10.1109/cisce69494.2026.11504731 · Cited by 15 · see citing works
“Splitwise [11] demonstrated the benefits of separating the prefill and decode phases across heterogeneous hardware.”
Yu, X., Jiang, D., Deng, J., et al. (2025) · Proceedings - International Symposium on Computer Architecture/Proceedings · doi:10.1145/3695053.3731016 · Cited by 11 · see citing works
“For larger-scale model training [38, 53], such as GPT-3 or LLaMA, the performance bottleneck is more complex, involving off-die memory bandwidth, inter-chip communication bandwidth, and distributed network communication bandwidth.”
Chimera: Communication Fusion for Hybrid Parallelism in Large Language Models
Qin, L., Cui, J., Cai, W., & Huang, J. (2025) · Proceedings - International Symposium on Computer Architecture/Proceedings · doi:10.1145/3695053.3731025 · Cited by 11 · see citing works
“We adopt hybrid parallelism with the configurations as shown in Due to the autoregressive nature of LLM inference, the process is divided into two phases: prefilling and decoding [48, 76].”
Kim, J., Shin, B., Chung, J., & Rhu, M. (2026) · IEEE International Symposium on High-Performance Computer Architecture (HPCA) · doi:10.1109/hpca68181.2026.11408569 · Cited by 11 · see citing works
“We explore this direction in Section IV-C, where we discuss the implications of serving AI agents over multiple queries with LLM request batching [32], [82], [98].”
An I/O Characterizing Study of Offloading LLM Models and KV Caches to NVMe SSD
Ren, Z., Doekemeijer, K., Matteis, T. D., Pinto, C., Stoica, R., & Trivedi, A. (2025) · Workshop on Challenges and Opportunities of Efficient and Performant Storage Systems (CHEOPS) · doi:10.1145/3719330.3721230 · Cited by 12 · see citing works
“Other products and service names might be trademarks of IBM or other companies.”
Wang, H., Wang, Z., Yue, Z., Long, Y., Wei, T., Yang, J., Wang, Y., Li, C., Wei, S., Hu, Y., et al. (2025) · IEEE/ACM International Symposium on Microarchitecture (MICRO) · doi:10.1145/3725843.3756037 · Cited by 11 · see citing works
A Water Efficiency Dataset for African Data Centers
Shumba, N., Tshekiso, O., Li, P., Fanti, G., & Ren, S. (2025) · Proceedings of the ACM SIGCAS/SIGCHI Conference on Computing and Sustainable Societies · doi:10.1145/3715335.3735483 · Cited by 10 · see citing works
“…the industry practice, since a fully-utilized system can lead to significant SLO violations, which are not tolerable in real-world LLM deployment, especially for commercial LLM applications such as real-time conversations that have strict SLO targets to deliver good quality of experiences [33, 45].”
InfluentialApt-Serve: Adaptive Request Scheduling on Hybrid Cache for Scalable LLM Inference Serving
Gao, S., Zhang, X., Shen, Y., & Chen, L. (2025) · Proceedings of the ACM on Management of Data · doi:10.1145/3725394 · Cited by 9 · see citing works
Dynamic Batch Processing with FlexiDecode Scheduler for Efficient LLM Inference in IIoT
Jia, X., Gu, B., Chen, J., Gao, L., Pang, W., Lv, G. D., Qu, Y., & Cui, L. (2025) · Big Data Mining and Analytics · doi:10.26599/bdma.2025.9020025 · Cited by 9 · see citing works
Understanding the Performance and Power of LLM Inferencing on Edge Accelerators
Arya, M., & Simmhan, Y. L. (2025) · IEEE International Parallel and Distributed Processing Symposium Workshops (IPDPSW) · doi:10.1109/ipdpsw66978.2025.00173 · Cited by 8 · see citing works
ClusterKV: Manipulating LLM KV Cache in Semantic Space for Recallable Compression
Liu, G., Li, C., Zhao, J., Zhang, C., & Guo, M. (2025) · ACM/IEEE Design Automation Conference (DAC) · doi:10.1109/dac63849.2025.11132479 · Cited by 8 · see citing works
“Since the autoregressive decoding stage is typically the performance bottleneck of LLM inference and is memory-bound [8], accessing the KV cache of all previous tokens results in increased latency for long-context inference.”
Self-citationCacheRoute: KV-Cache-Aware Routing for LLM Backend Services on Kubernetes
Pi, S., Zhang, S., & Shawulieti, S. (2026) · International Conference on Electronic Information Technology and Computer Engineering (EITCE) · doi:10.1109/eitce70137.2026.11634378 · Cited by 8 · see citing works
“The separation of prefill and decode execution stages reveals substantial heterogeneity in computational behavior, memory consumption, and latency sensitivity, motivating scheduling mechanisms that operate beyond conventional request-level load balancing [16].”
Zhang, C., You, Y., Wang, N., Park, J., & Zhang, L. (2025) · IEEE Journal on Emerging and Selected Topics in Circuits and Systems · doi:10.1109/jetcas.2025.3575272 · Cited by 8 · see citing works
Self-citationDeploying Foundation Model Powered Agent Services: A Survey
Xu, W., Chen, J., Zheng, P., Yi, X., Tian, T., Zhu, W., Wan, Q., Wang, H., Fan, Y., Su, Q., et al. (2025) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2025.3580745 · Cited by 16 · see citing works
“Reference [254] proposes Splitwise, a technique that schedules the prompt computation and token generation phases to different machines, optimizing hardware utilization and overall efficiency.”
Tan, X., Li, B., Qiu, X., Qu, C., Chu, W., Xu, Y., & Qi, Y. (2025) · International World Wide Web Conference (WWW) · doi:10.1145/3701716.3715247 · Cited by 7 · see citing works
“For complex tasks, LLM-based agents often require lengthy token generation to execute effective planning, which extends TTFT and RT and may hinder usability in time-sensitive applications [19].”
Good things come in small packages: Should we build AI clusters with Lite-GPUs?
Canakci, B., Liu, J., Wu, X., Cheriere, N., Costa, P., Legtchenko, S., Narayanan, D., & Rowstron, A. I. T. (2025) · Workshop on Hot Topics in Operating Systems · doi:10.1145/3713082.3730390 · Cited by 6 · see citing works
“In the evaluation, we assume that different phases can execute on different Lite-GPU clusters [40, 63] to demonstrate the hardware benefits achievable with Lite-GPUs.”
InfluentialELLIE: Energy-Efficient LLM Inference at the Edge Via Prefill-Decode Splitting
Fan, H., Lin, Y.-C., & Prasanna, V. K. (2025) · IEEE International Conference on Application-specific Systems, Architectures and Processors (ASAP) · doi:10.1109/asap65064.2025.00031 · Cited by 6 · see citing works
“Splitwise [13] splits LLM inference into Prefill and Decode stages executed on different machines to improve cost-efficiency, leveraging powerful yet expensive GPUs for the Prefill phase and cost-efficient GPUs for the Decode phase.”
SAGESERVE: Optimizing LLM Serving on Cloud Data Centers with Forecast Aware Auto-Scaling
Jaiswal, S., Jain, K., Simmhan, Y. L., Parayil, A., Mallick, A., Wang, R., Amant, R. S., Bansal, C., Rühle, V., Kulkarni, A., et al. (2025) · Proceedings of the ACM on Measurement and Analysis of Computing Systems · doi:10.1145/3771576 · Cited by 6 · see citing works
“SageServe is validated using a realistic simulation harness that we have developed by extending SplitWise [35], using 4 popular open-source LLMs, for week-long traces from multiple US data centers across two time periods, and also compared against Chiron [36], a State of the Art (SOTA) baseline.”
InfluentialQiu, H., Biswas, A., Zhao, Z., Mohan, J., Khare, A., Choukse, E., Goiri, Í., Zhang, Z., Shen, H., Bansal, C., et al. (2025) · ACM Symposium on Cloud Computing (SoCC) · doi:10.1145/3772052.3772254 · Cited by 6 · see citing works
Self-citationCai, J., Wang, X., Gao, M., et al. (2025) · Proceedings - International Symposium on High-Performance Computer Architecture/Proceedings · doi:10.1109/hpca61900.2025.00048 · Cited by 6 · see citing works
“2) Workload: To comprehensively evaluate SoMa effects and analyze the trade-offs behind different DRAM communication scheduling schemes, we scale batch size from 1 to 64, covering latency-sensitive scenarios to throughput-centric scenarios as introduced in Sec.”
Hermes: Algorithm-System Co-design for Efficient Retrieval-Augmented Generation At-Scale
Shen, M., Umar, M., Maeng, K., Suh, G. E., & Gupta, U. (2025) · Proceedings - International Symposium on Computer Architecture/Proceedings · doi:10.1145/3695053.3731076 · Cited by 5 · see citing works
“Our baseline assumes an input sequence length of 512 tokens and a generated output of 256 tokens, based on an average short length of queries and responses as seen in production systems [35].”
Characterizing and Optimizing LLM Inference Workloads on CPU-GPU Coupled Architectures
Vellaisamy, P., Labonte, T., Chakraborty, S., Turner, M., Sury, S., & Shen, J. P. (2025) · IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS) · doi:10.1109/ispass64960.2025.00015 · Cited by 6 · see citing works
“The prefill stage puts pressure on the compute resources, while the decode stage puts pressure on the memory subsystems [34] [35].”
A Case Study of Environmental Footprints for Generative AI Inference: Cloud versus Edge
Li, P., Islam, M. J., & Ren, S. (2025) · ACM SIGMETRICS Performance Evaluation Review · doi:10.1145/3764944.3764950 · Cited by 5 · see citing works
“When deploying generative AI models in the cloud, further quantization is typically avoided to preserve optimal model performance, especially for models with small and medium sizes [12, 13].”
Liu, M., Gu, J., & Zhao, T. (2025) · IEEE International Conference on Web Services (ICWS) · doi:10.1109/icws67624.2025.00045 · Cited by 5 · see citing works
“Workloads are generated using the Azure LLM Server-less Trace [1], a representative serverless LLM benchmark.”
InfluentialInput-Dependent Power Usage in GPUs
Gregersen, T., Patel, P., & Choukse, E. (2024) · doi:10.1109/scw63240.2024.00235 · Cited by 5 · see citing works
“Research on reducing accelerator peak power draw or energy often focuses on hardware design and the surrounding infrastructure: system design and scheduling [14], [15], [20], [21], power control [9], [22], or efficient chips [23]–[25].”
Self-citationPerformance Characterization of Expert Router for Scalable LLM Inference
Pichlmeier, J., Ross, P., & Luckow, A. (2024) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata62323.2024.10826121 · Cited by 8 · see citing works
“Previous work has investigated how factors such as the number of input tokens, batch size and number of concurrent users can affect these metrics [33]–[35].”
Data+AI: LLM4Data and Data4LLM
Li, G., Wang, J., Zhang, C., & Wang, J. (2025) · ACM SIGMOD International Conference on Management of Data (SIGMOD) · doi:10.1145/3722212.3725641 · Cited by 5 · see citing works
“They propose two-stage disaggregation [25, 44, 45, 69], which splits prefill and decoding computation on different GPUs.”
Self-citationCOMET: Towards Practical W4A4KV4 LLMs Serving
Liu, L., Cheng, L., Ren, H., Zhaohui, X., Pan, Y., Wang, M., Li, X., Han, Y., & Wang, Y. (2025) · International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) · doi:10.1145/3676641.3716252 · Cited by 5 · see citing works
“Explore the performance gains from different optimization strategies.”
Near-Memory LLM Inference Processor based on 3D DRAM-to-logic Hybrid Bonding
Han, S., Yoon, B., Park, G., Song, C., Kim, D., & Kim, J.-J. (2025) · ACM/IEEE Design Automation Conference (DAC) · doi:10.1109/dac63849.2025.11132870 · Cited by 7 · see citing works
“The size of KV cache increases with the batch size linearly so that the batch size is typically limited to less than 64 in real LLM inference servings [25].”
Disaggregated Speculative Decoding for Carbon-Efficient LLM Serving
Shi, T., Wu, Y., Liu, S., & Ding, Y. (2025) · IEEE Computer Architecture Letters · doi:10.1109/lca.2025.3630094 · Cited by 4 · see citing works
“For DPD, we leveraged community implementations of vLLM based on the design principles of Splitwise [1].”
Green prompt engineering for sustainable generative AI
Podder, S., Date, H., & Murthy, S. (2026) · Environmental Science and Ecotechnology · doi:10.1016/j.ese.2026.100684 · Cited by 4 · see citing works
“These two phases tend to be computationally and memory-intensive, respectively [26].”
m2LLM: A Multi-Dimensional Optimization Framework for LLM Inference on Mobile Devices
Liu, K., Zhou, X., & Li, L. (2025) · IEEE Transactions on Parallel and Distributed Systems · doi:10.1109/tpds.2025.3587445 · Cited by 6 · see citing works
“For instance, distributed inference [10], [11], offloading [12], [13], [14], batch optimization [15], [16] have been proposed to optimize the inference process on the cloud environment.”
Efficient LLM Inference via Chunked Prefills
Agrawal, A., Kedia, N., Panwar, A., Mohan, J., Kwatra, N., Gulavani, B. S., Tumanov, A., & Ramjee, R. (2025) · ACM SIGOPS Operating Systems Review · doi:10.1145/3759441.3759444 · Cited by 4 · see citing works
“Similar to Patel et al. [21], to account for the intrinsic performance limitations of a model and hardware pair, we define the SLO on P99 TBT to be equal to 5 x and 25 x the execution time of a decode iteration for a request (with prefilliength of 4k and 32 batch size) running without any prefill…”
Hetis: Serving LLMs in Heterogeneous GPU Clusters with Fine-grained and Dynamic Parallelism
Mo, Z., Liao, J., Xu, H., Zhou, Z., & Xu, C. (2025) · International Conference for High Performance Computing, Networking, Storage and Analysis (SC) · doi:10.1145/3712285.3759784 · Cited by 4 · see citing works
“For instance, the Prefill-Decode disaggregation scheme like Splitwise [35] consumes substantial memory by hosting extra copies of model parameters.”
InfluentialSlower is Greener: Acceptance of Eco-feedback Interventions on Carbon Heavy Internet Services
Kim, H., Young, S., Chen, X., Gupta, U., & Hester, J. D. (2025) · ACM Journal on Computing and Sustainable Societies · doi:10.1145/3723038 · Cited by 4 · see citing works
“Ongoing work also explores designing power and energy efficient servers for salient applications like AI [25, 31].”
Optimizing LLM Inference: Fluid-Guided Online Scheduling with Memory ConstraintsPreprint
Ao, R., Luo, G., Simchi‐Levi, D., & Wang, X. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5195463 · Cited by 4 · see citing works
BATON: Enhancing Batch-wise Inference Efficiency for Large Language Models via Dynamic Re-batching
Cong, P., Chen, Q., Zhao, H., & Yang, T. (2025) · International World Wide Web Conference (WWW) · doi:10.1145/3696410.3714950 · Cited by 3 · see citing works
“Moreover, the inference process can be divided into two phases: prefilling and decoding [15, 16].”
Cannikin: No Lagger of SLO in Concurrent Multiple LoRA LLM Serving
Zhu, R., Jiang, Z., Zhang, Z., Liu, X., Liu, X., & Jin, X. (2025) · IEEE Transactions on Parallel and Distributed Systems · doi:10.1109/tpds.2025.3590014 · Cited by 3 · see citing works
“Distserve [15] and SplitWise [16] disaggregate prefill phases and decoding phases into different instances to improve the system efficiency and cost.”
InfluentialS2M3: Split-and-Share Multi-Modal Models for Distributed Multi-Task Inference on the Edge
Yoon, J., Lee, J., He, T., et al. (2025) · IEEE International Conference on Distributed Computing Systems (ICDCS) · doi:10.1109/icdcs63083.2025.00089 · Cited by 3 · see citing works
“…✓ ✗ LLM-Pruner [8], SparseGPT [9], LGViT [33], PowerInfer [34] Inference Lightweight Model ✗ ✗ DINA [16], A3C [17], PETALS [18], Splitwise [19], EdgeShard [20] Inference Intra-Module Partitioning ✗ ✗ We design S2M3 , a S plit-and-S hare M ulti-M odal M odel architecture for multi-task…”
LibraPIM: Dynamic Load Rebalancing to Maximize Utilization in PIM-Assisted LLM Inference Systems
Cho, H., Jang, Y., Kim, H., et al. (2025) · International Conference on Parallel Architectures and Compilation Techniques · doi:10.1109/pact65351.2025.00016 · Cited by 3 · see citing works
“LLM inference typically relies on compute accelerators, such as GPUs [11, 12] and NPUs [13, 14, 15, 16, 17, 18, 19, 20], which exploit massive parallelism to accelerate computationally intensive layers.”
BucketServe: Bucket-Based Dynamic Batching for Smart and Efficient LLM Inference Serving
Zheng, W., Xu, M., Song, S. H., & Ye, K. (2025) · IEEE International Conference on Internet of Things (iThings) and IEEE Green Computing and Communications (GreenCom) and IEEE Cyber, Physical and Social Computing (CPSCom) and IEEE Smart Data (SmartData) · doi:10.1109/ithings-greencom-cpscom-smartdata-cybermatics67059.2025.00041 · Cited by 3 · see citing works
Zheng, T., Bi, Y., Han, G., et al. (2026) · IEEE Transactions on Mobile Computing · doi:10.1109/tmc.2026.3650838 · Cited by 3 · see citing works
“[ 27] introduces Splitwise, a model deployment and scheduling method that splits the prefill and decoding phases across separate devices.”
Sometimes Painful but Promising: Feasibility and Trade-Offs of On-Device Language Model Inference
Abstreiter, M., Tarkoma, S., & Morabito, R. (2026) · ACM Transactions on Embedded Computing Systems · doi:10.1145/3788870 · Cited by 7 · see citing works
“Despite having 8 GB of nominal memory, the usable memory was limited to below 7 GB, as the operating system and background processes collectively occupied over 1 GB.”
GPU-Centric Memory Tiering for LLM Serving With NVIDIA Grace Hopper Superchip
Choi, W.-H., Jeong, J., Jang, H., & Ahn, J. (2025) · IEEE Computer Architecture Letters · doi:10.1109/lca.2025.3533588 · Cited by 3 · see citing works
Topology-Aware Virtualization over Inter-Core Connected Neural Processing Units
Feng, D., Feng, E., Du, D., et al. (2025) · Proceedings - International Symposium on Computer Architecture/Proceedings · doi:10.1145/3695053.3731002 · Cited by 3 · see citing works
FlexNeRFer: A Multi-Dataflow, Adaptive Sparsity-Aware Accelerator for On-Device NeRF Rendering
Noh, S.-H., Shin, B., Choi, J., et al. (2025) · Proceedings - International Symposium on Computer Architecture/Proceedings · doi:10.1145/3695053.3731107 · Cited by 3 · see citing works
“As the demand for Transformer-based LLMs continues to grow across various real-world applications, research efforts in both academia and industry have been actively directed toward improving inference speed [20, 23, 30, 43, 56, 57, 63, 66, 73, 97].”
Chameleon: Adaptive Caching and Scheduling for Many-Adapter LLM Inference Environments
Iliakopoulou, N., Stojkovic, J., Alverti, C., Xu, T., Franke, H., & Torrellas, J. (2025) · IEEE/ACM International Symposium on Microarchitecture (MICRO) · doi:10.1145/3725843.3756083 · Cited by 5 · see citing works
Engine-Agnostic Model Hot-Swapping for Cost-Effective LLM Inference
Stoyanov, R., Spišaková, V., Reber, A., Armour, W., Copik, M., & Bruno, R. (2025) · International Conference for High Performance Computing, Networking, Storage and Analysis (SC) · doi:10.1145/3731599.3767354 · Cited by 2 · see citing works
“This overhead far exceeds the latency requirements for token generation, with industry target for high-quality interactive experience typically below the average human reaction time in the range of 200-250 ms [41].”
Guo, T., Zhang, X.-W., Du, J.-S., Chen, Z.-G., Xiao, N., & Lu, Y.-T. (2025) · International Conference on Software Composition · doi:10.1145/3712285.3759823 · Cited by 15 · see citing works
“Finally, a sampling strategy (e.g., greedy, top-k, or nucleus sampling) is applied to select the next token for generation.”
Tropical: Enhancing SLO Attainment in Disaggregated LLM Serving via SLO-Aware Multiplexing
Ma, J., Chen, J., Li, X., Duan, J., Duanmu, H., Zhang, X., Yang, C., & Lin, D. (2025) · ACM/IEEE Design Automation Conference (DAC) · doi:10.1109/dac63849.2025.11132617 · Cited by 2 · see citing works
“Similar conclusions can also be drawn from the dataset evaluated in other LLM serving-related works [7].”
A Deployment-Aware Framework for Carbon- and Water- Efficient LLM Serving
Hoxha, J., Thanasi-Boçe, M., & Khalifa, T. (2025) · Sustainability · doi:10.3390/su172310473 · Cited by 2 · see citing works
M etis : A Non-Clairvoyant, Workflow-Aware OS Scheduler for Serverless Applications
Tang, W., Yang, Y., & Wu, J. (2025) · ACM Symposium on Cloud Computing (SoCC) · doi:10.1145/3772052.3772221 · Cited by 2 · see citing works
“These serverless work-flows, represented as directed acyclic graphs (DAGs), have gained significant adoption across diverse domains including video processing pipelines [3, 53], machine learning inference chains [16, 37], and large-scale data analytics [40].”
Oneiros: KV Cache Optimization through Parameter Remapping for Multi-tenant LLM Serving
Li, R., Pal, S., Pullu, V. N., Sinha, P., Ryoo, J., John, L. K., & Yadwadkar, N. J. (2025) · ACM Symposium on Cloud Computing (SoCC) · doi:10.1145/3772052.3772215 · Cited by 2 · see citing works
“Traces: We evaluate Oneiros using the Azure coding LLM inference traces that capture real-world query arrival patterns characterized by bursty query patterns [46, 60].”
InfluentialAdaptive Request Scheduling and Load Balancing for Edge Deployed Large Language Models
Mou, F., Tang, Z., Jia, W., & Zhao, W. (2026) · IEEE Transactions on Services Computing · doi:10.1109/tsc.2026.3665250 · Cited by 2 · see citing works
“Recent resource-aware edge inference systems have explored distributed LLM serving at the edge [19], [20].”
RPU – A Reasoning Processing Unit
Adiletta, M. J., Wei, G.-Y., & Brooks, D. M. (2026) · IEEE International Symposium on High-Performance Computer Architecture (HPCA) · doi:10.1109/hpca68181.2026.11408490 · Cited by 2 · see citing works
“The first step toward low-latency LLM inference is to separate prefill and decode onto different systems, as in Dynamo [17] and the Splitwise execution model [50].”
VectorLiteRAG: Latency-Aware and Fine-Grained Resource Partitioning for Efficient RAG
Kim, J., & Mahajan, D. (2026) · IEEE International Symposium on High-Performance Computer Architecture (HPCA) · doi:10.1109/hpca68181.2026.11408552 · Cited by 9 · see citing works
“This allows in-stantiating a Beta distribution with inferred mean and variance for any cache coverage configuration.”
AUM: Unleashing the Efficiency Potential of Shared Processors with Accelerator Units for LLM Serving
Wang, X., Li, C., Zhuansun, Y., Guo, J., Hou, X., Wang, J., Wang, L., Chen, W., Huang, C., Yang, G., et al. (2026) · IEEE International Symposium on High-Performance Computer Architecture (HPCA) · doi:10.1109/hpca68181.2026.11408539 · Cited by 2 · see citing works
“Firstly, AU-enabled CPU should be used as prompt machines under specific scenarios [67].”
InfluentialBlitzScale: Fast and Live Large Model Autoscaling with O(1) Host Caching
Zhang, D., Liu, Y., Wang, H., Wei, X., Shan, Y., Chen, R., & Chen, H. Z. (2026) · ACM Transactions on Computer Systems · doi:10.1145/3807606 · Cited by 2 · see citing works
“Because the scaling requirements are closely related to the incoming request rates, we chose three typical real-world traces: BurstGPT [71], Azure-Code and AzureConv, both from Azure [55].”
InfluentialSLICE: S LO-Driven Scheduling for L LM I nference on E dge Computing Devices
Zhou, P., Lei, Y., Liu, L., Xu, X., Yu, H., Sun, G., Ergu, D., & Dai, Y. (2026) · IEEE Transactions on Services Computing · doi:10.1109/tsc.2026.3699152 · Cited by 2 · see citing works
“Splitwise[32]decoupledprefillingand decodingbatchestoreduceinterference.”
Wei, Y., Cai, J., Gao, M., et al. (2025) · Proceedings - ACM IEEE Design Automation Conference · doi:10.1109/dac63849.2025.11132440 · Cited by 2 · see citing works
“To further investigate the architecture and layer-pipeline mapping of many-core accelerators, we conduct a DSE, providing in-depth analysis and discussion on the two critical factors of mapping and architecture: pipeline depth and buffer size.”
LLMShare: Optimizing LLM Inference Serving with Hardware Architecture Exploration
Liu, H., Bai, C., Xu, P., Yin, L., Yu, X. D., Zhen, H., Yuan, M., Ho, T.-Y., & Bei, Y. (2025) · ACM/IEEE Design Automation Conference (DAC) · doi:10.1109/dac63849.2025.11132534 · Cited by 2 · see citing works
“For instance, the prefill stage is computation-intensive and requires limited batch sizes to maintain performance, whereas the decoding phase can handle larger batch sizes, enabling higher throughput [17].”
InfluentialLiakopoulos, D., Sinha, P., Hu, T., Lee, M., & Yadwadkar, N. J. (2025) · International Conference for High Performance Computing, Networking, Storage and Analysis (SC) · doi:10.1145/3712285.3759867 · Cited by 2 · see citing works
“Several works focus on optimizing request scheduling [2, 3, 7, 22, 55, 67, 81, 88, 104, 107], reconfiguration [94], and speculative execution [56, 98].”
InfluentialZhang, Z., Shen, H., Vargaftik, S., Basat, R. B., Mitzenmacher, M., & Yu, M. (2025) · Proceedings of the ACM SIGCOMM 2025 Conference · doi:10.1145/3718958.3750481 · Cited by 2 · see citing works
DroidSpeak: KV Cache Sharing for Cross-LLM Communication and Multi-LLM Serving
Liu, Y., Huang, Y., Yao, J., Feng, S., Gu, Z., Du, K., Li, H., Cheng, Y., Jiang, J., Lu, S., et al. (2024) · arXiv · doi:10.48550/arxiv.2411.02820 · Cited by 2 · see citing works
Self-citationPreble: Efficient Distributed Prompt Scheduling for LLM Serving
Srivatsa, V., He, Z., Abhyankar, R., Li, D., & Zhang, Y. (2024) · arXiv · doi:10.48550/arxiv.2407.00023 · Cited by 1 · see citing works
“It supports both data parallelism and model parallelism.”
BackgroundMethodOnline Scheduling for LLM Inference with KV Cache Constraints
Jaillet, P., Jiang, J., Mellou, K., Molinaro, M., Podimata, C., & Zhou, Z. (2025) · arXiv · doi:10.48550/arxiv.2502.07115 · Cited by 1 · see citing works
“While many studies have focused on improving LLM inference efficiency via engineering [Agrawal et al., 2023, Kwon et al., 2023, Patel et al., 2023, Pope et al., 2023, Sheng et al., 2023, Yu et al., 2022, Zhong et al., 2024], there are very few formal models in this space.”
- Authors
- Patel, P., Choukse, E., Zhang, C., Shah, A., Goiri, Í., Maleki, S., & Bianchini, R.
- Affiliations
- University of Washington (US)
- Microsoft (United States) (US)
- ORCID
- Venue
- 2024 ACM/IEEE 51st Annual International Symposium on Computer Architecture (ISCA)
- Publisher
- IEEE
- Published
- 2024-06
- Pages
- 118-132
- ISSN
- 1063-6897 (ISSN-L)
- OpenAlex
- W4401211704
- Semantic Scholar
- ad9146d98ae95bbeeef460abe083ecc2c4798672
- Cited by
- 298
- References
- 59
- Citations by year
- 2026: 176 · 2025: 108 · 2024: 14
- FWCI
- 90.63
- Type
- conference paper
- Language
- en
- Open access
- closed
- Licence
- Locations
- Proceedings - International Symposium on Computer Architecture/Proceedings (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Hardware and Architecture › Parallel Computing and Optimization Techniques
- Physical Sciences › Computer Science › Information Systems › Cloud Computing and Resource Management
- Physical Sciences › Computer Science › Computer Networks and Communications › Distributed and Parallel Computing Systems
- Keywords
- large language model inference, phase splitting, token generation, GPU clusters, request scheduling, resource management, power efficiency, cost efficiency, throughput, network libraries
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
