LoongServe: Efficiently Serving Long-Context Large Language Models with Elastic Sequence Parallelism
Open accessAbstract
The context window of large language models (LLMs) is rapidly increasing, leading to a huge variance in resource usage between different requests as well as between different phases of the same request. Restricted by static parallelism strategies, existing LLM serving systems cannot efficiently utilize the underlying resources to serve variable-length requests in different phases. To address this problem, we propose a new parallelism paradigm, elastic sequence parallelism (ESP), to elastically adapt to the variance across different requests and phases. Based on ESP, we design and build LoongServe, an LLM serving system that (1) improves computation efficiency by elastically adjusting the degree of parallelism in real-time, (2) improves communication efficiency by reducing key-value cache migration overhead and overlapping partial decoding communication with computation, and (3) improves GPU memory efficiency by reducing key-value cache fragmentation across instances. Our evaluation under diverse real-world datasets shows that LoongServe improves the throughput by up to 3.85× compared to chunked prefill and 5.81× compared to prefill-decoding disaggregation.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Helix [21] optimizes placement on heterogeneous single-cluster GPUs; LoongServe [48] reshapes parallelism for long contexts; ServerlessLLM [47] attacks cold-start latency, a bottleneck we observe in S4 warmup.
“Helix [21] optimizes placement on heterogeneous single-cluster GPUs; LoongServe [48] reshapes parallelism for long contexts; ServerlessLLM [47] attacks cold-start latency, a bottleneck we observe in S4 warmup.”
From Janus: Joint Prefill/Decode Disaggregation with KV-Cache-Aware Multi-Cloud Routing for Edge-Adjacent LLM Serving (2026) · cites this paper as [48]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Optimus: an efficient dynamic resource scheduler for deep learning clusters
Peng, Y.-H., Bao, Y., Chen, Y.-R., Wu, C., & Guo, C.-X. (2018) · European Conference on Computer Systems · doi:10.1145/3190508.3190517 · Cited by 544 · see citing works
GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints
Ainslie, J., Lee-Thorp, J., de Jong, M., Zemlyanskiy, Y., Lebron, F., & Sanghai, S. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.298 · Cited by 454 · see citing works
GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints
Ainslie, J., Lee-Thorp, J., de Jong, M., Zemlyanskiy, Y., Lebrón, F., & Sanghai, S. (2023) · arXiv · doi:10.48550/arxiv.2305.13245 · Cited by 447 · see citing works
Triton: an intermediate language and compiler for tiled neural network computations
Tillet, P., Kung, H. T., & Cox, D. (2019) · Proceedings of the 3rd ACM SIGPLAN International Workshop on Machine Learning and Programming Languages · doi:10.1145/3315508.3329973 · Cited by 318 · see citing works
Efficient dynamic programming using quadrangle inequalities
Yao, F. F. (1980) · ACM Symposium on Theory of Computing (STOC) · doi:10.1145/800141.804691 · Cited by 134 · see citing works
Sequence Parallelism: Long Sequence Training from System Perspective
Li, S., Xue, F., Baranwal, C., Li, Y., & You, Y. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.134 · Cited by 88 · see citing works
System Optimizations for Enabling Training of Extreme Long Sequence Transformer Models
Jacobs, S. A., Tanaka, M., Zhang, C., Zhang, M., Aminabadi, R. Y., Song, S. L., Rajbhandari, S., & He, Y. (2024) · IEEE International Symposium on Parallel & Distributed Processing, Workshops and Phd Forum · doi:10.1109/ipdpsw63119.2024.00208 · Cited by 17 · see citing works
Flagged first, then by citations
Research works printed but not linked: 45
- Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. GPT-4 technical report. In arXiv.
- Muhammad Adnan, Akhil Arunkumar, Gaurav Jain, Prashant J Nair, Ilya Soloveychik, and Purushotham Kamath. 2024. Keyformer: KV Cache Reduction through Key Tokens Selection for Efficient Genera- tive Inference. In Conference on Machine Learning and Systems.
- Amey Agrawal, Ashish Panwar, Jayashree Mohan, Nipun Kwatra, Bhar- gav S. Gulavani, and Ramachandran Ramjee. 2023. SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefills. In arXiv.
- Chenxin An, Shansan Gong, Ming Zhong, Xingjian Zhao, Mukai Li, Jun Zhang, Lingpeng Kong, and Xipeng Qiu. 2023. L-Eval: Instituting Standardized Evaluation for Long Context Language Models. In arXiv.
- William Brandon, Aniruddha Nrusimha, Kevin Qian, Zachary Ankner, Tian Jin, Zhiye Song, and Jonathan Ragan-Kelley. 2023. Striped Atten- tion: Faster Ring Attention for Causal Transformers. In arXiv.
- Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. 2019. Gen- erating Long Sequences with Sparse Transformers. In arXiv.
- Tri Dao. 2023. FlashAttention-2: Faster Attention with Better Paral- lelism and Work Partitioning. In arXiv.
- Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. 2022. FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness. In Neural Information Processing Systems.
- Diandian Gu, Yihao Zhao, Yinmin Zhong, Yifan Xiong, Zhenhua Han, Peng Cheng, Fan Yang, Gang Huang, Xin Jin, and Xuanzhe Liu. 2023. ElasticFlow: An Elastic Serverless Training Platform for Distributed Deep Learning. In ACM ASPLOS.
- Juncheng Gu, Mosharaf Chowdhury, Kang G Shin, Yibo Zhu, Myeong- jae Jeon, Junjie Qian, Hongqiang Harry Liu, and Chuanxiong Guo. 2019. Tiresias: A GPU Cluster Manager for Distributed Deep Learning.. In USENIX NSDI.
- Connor Holmes, Masahiro Tanaka, Michael Wyatt, Ammar Ahmad Awan, Jeff Rasley, Samyam Rajbhandari, Reza Yazdani Aminabadi, Heyang Qin, Arash Bakhtiari, Lev Kurilenko, and Yuxiong He. 2024. DeepSpeed-FastGen: High-throughput Text Generation for LLMs via MII and DeepSpeed-Inference. In arXiv.
- Cunchen Hu, Heyang Huang, Liangliang Xu, Xusheng Chen, Jiang Xu, Shuang Chen, Hao Feng, Chenxi Wang, Sa Wang, Yungang Bao, Ninghui Sun, and Yizhou Shan. 2024. Inference without Interference: Disaggregate LLM Inference for Mixed Downstream Workloads. In arXiv.
- Changho Hwang, Taehyun Kim, Sunghyun Kim, Jinwoo Shin, and KyoungSoo Park. 2021. Elastic Resource Sharing for Distributed Deep Learning. In USENIX NSDI.
- Suhas Jayaram Subramanya, Daiyaan Arfeen, Shouxu Lin, Aurick Qiao, Zhihao Jia, and Gregory R. Ganger. 2023. Sia: Heterogeneity-aware, goodput-optimized ML-cluster scheduling. In ACM SOSP.
- Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guil- laume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie-Anne Lachaux, Pierre Stock, Sandeep Subramanian, Sophia Yang, Szymon Antoniak, Teven Le Scao, Théophile Gervet, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. 2024. Mixtral of Experts. In arXiv.
- Vijay Korthikanti, Jared Casper, Sangkug Lym, Lawrence McAfee, Michael Andersch, Mohammad Shoeybi, and Bryan Catanzaro. 2022. Reducing Activation Recomputation in Large Transformer Models. In arXiv.
- Vijay Anand Korthikanti, Jared Casper, Sangkug Lym, Lawrence McAfee, Michael Andersch, Mohammad Shoeybi, and Bryan Catan- zaro. 2023. Reducing activation recomputation in large transformer models. In Conference on Machine Learning and Systems.
- Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient memory management for large language model serving with pagedattention. In ACM SOSP.
- Dacheng Li, Rulin Shao, Anze Xie, Eric P Xing, Joseph E Gonzalez, Ion Stoica, Xuezhe Ma, and Hao Zhang. 2023. Lightseq: Sequence level parallelism for distributed training of long context transformers. In arXiv.
- Mingzhen Li, Wencong Xiao, Biao Sun, Hanyu Zhao, Hailong Yang, Shiru Ren, Zhongzhi Luan, Xianyan Jia, Yi Liu, Yong Li, Wei Lin, and Depei Qian. 2023. EasyScale: Accuracy-consistent Elastic Training for Deep Learning. In ACM SC.
- Zhuohan Li, Lianmin Zheng, Yinmin Zhong, Vincent Liu, Ying Sheng, Xin Jin, Yanping Huang, Zhifeng Chen, Hao Zhang, Joseph E Gonzalez, et al. 2023. AlpaServe: Statistical Multiplexing with Model Parallelism for Deep Learning Serving. In USENIX OSDI.
- Bin Lin, Tao Peng, Chen Zhang, Minmin Sun, Lanbo Li, Hanyu Zhao, Wencong Xiao, Qi Xu, Xiafei Qiu, Shen Li, Zhigang Ji, Yong Li, and Wei Lin. 2024. Infinite-LLM: Efficient LLM Service for Long Context with DistAttention and Distributed KVCache. In arXiv.
- Hao Liu, Wilson Yan, Matei Zaharia, and Pieter Abbeel. 2024. World Model on Million-Length Video and Language with RingAttention. In arXiv.
- Hao Liu, Matei Zaharia, and Pieter Abbeel. 2023. Ring Attention with Blockwise Transformers for Near-Infinite Context. In arXiv.
- Xiaoxuan Liu, Cade Daniel, Langxiang Hu, Woosuk Kwon, Zhuohan Li, Xiangxi Mo, Alvin Cheung, Zhijie Deng, Ion Stoica, and Hao Zhang. 2024. Optimizing Speculative Decoding for Serving Large Language Models Using Goodput. In arXiv.
- Yujing Ma, Florin Rusu, Kesheng Wu, and Alexander Sim. 2021. Adap- tive Elastic Training for Sparse Deep Learning on Heterogeneous Multi-GPU Servers. In arXiv.
- Philipp Moritz, Robert Nishihara, Stephanie Wang, Alexey Tumanov, Richard Liaw, Eric Liang, Melih Elibol, Zongheng Yang, William Paul, Michael I. Jordan, and Ion Stoica. 2018. Ray: A Distributed Framework for Emerging AI Applications. In USENIX OSDI.
- Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong. 2023. CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesis. In International Conference on Learning Representations (ICLR).
- Matteo Pagliardini, Daniele Paliotta, Martin Jaggi, and François Fleuret. 2023. Faster Causal Attention Over Large Sequences Through Sparse Flash Attention. In arXiv.
- Pratyush Patel, Esha Choukse, Chaojie Zhang, Íñigo Goiri, Aashaka Shah, Saeed Maleki, and Ricardo Bianchini. 2023. Splitwise: Efficient generative LLM inference using phase splitting. In arXiv.
- Archit Patke, Dhemath Reddy, Saurabh Jha, Haoran Qiu, Christian Pinto, Shengkun Cui, Chandra Narayanaswami, Zbigniew Kalbarczyk, and Ravishankar Iyer. 2024. One Queue Is All You Need: Resolving Head-of-Line Blocking in Large Language Model Serving. In arXiv.
- Aurick Qiao, Sang Keun Choe, Suhas Jayaram Subramanya, Willie Neiswanger, Qirong Ho, Hao Zhang, Gregory R. Ganger, and Eric P. Xing. 2021. Pollux: Co-adaptive Cluster Scheduling for Goodput- Optimized Deep Learning. In USENIX OSDI.
- Noam Shazeer. 2019. Fast Transformer Decoding: One Write-Head is All You Need. In arXiv.
- Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. 2019. Megatron-LM: Training Multi-billion Parameter Language Models using Model Parallelism. In arXiv.
- Weigao Sun, Zhen Qin, Dong Li, Xuyang Shen, Yu Qiao, and Yiran Zhong. 2024. Linear Attention Sequence Parallelism. In arXiv.
- Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie- Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. LLaMA: Open and Efficient Foundation Language Models. In arXiv.
- Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Alma- hairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. In arXiv.
- Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. At- tention is all you need. In Neural Information Processing Systems.
- Bingyang Wu, Yinmin Zhong, Zili Zhang, Gang Huang, Xuanzhe Liu, and Xin Jin. 2023. Fast Distributed Inference Serving for Large Language Models. In arXiv.
- Bingyang Wu, Ruidong Zhu, Zili Zhang, Peng Sun, Xuanzhe Liu, and Xin Jin. 2024. dLoRA: Dynamically Orchestrating Requests and Adapters for LoRA LLM Serving. In USENIX OSDI.
- Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. 2024. Efficient Streaming Language Models with Attention Sinks. In International Conference on Learning Representations (ICLR).
- Gyeong-In Yu, Joo Seong Jeong, Geon-Woo Kim, Soojeong Kim, and Byung-Gon Chun. 2022. Orca: A Distributed Serving System for Transformer-BasedGenerative Models. In USENIX OSDI.
- Tao Yuan, Xuefei Ning, Dong Zhou, Zhijie Yang, Shiyao Li, Minghui Zhuang, Zheyue Tan, Zhuyu Yao, Dahua Lin, Boxun Li, Guohao Dai, Shengen Yan, and Yu Wang. 2024. LV-Eval: A Balanced Long-Context Benchmark with 5 Length Levels Up to 256K. In arXiv.
- Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher Ré, Clark Barrett, et al. 2024. H2o: Heavy-hitter oracle for efficient generative inference of large language models. In Neural Information Processing Systems.
- Yinmin Zhong, Shengyu Liu, Junda Chen, Jianbo Hu, Yibo Zhu, Xu- anzhe Liu, Xin Jin, and Hao Zhang. 2024. DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving. In USENIX OSDI.
Cited by
Sorted by citations
CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusion
Yao, J., Li, H., Liu, Y., Ray, S., Cheng, Y., Zhang, Q., Du, K., Lu, S., & Jiang, J. (2025) · Proceedings of the Twentieth European Conference on Computer Systems · doi:10.1145/3689031.3696098 · Cited by 75 · see citing works
Large models for intelligent transportation systems and autonomous vehicles: A survey
Gan, L., Chu, W., Li, G., Tang, X., & Li, K. (2024) · Advanced Engineering Informatics · doi:10.1016/j.aei.2024.102786 · Cited by 79 · see citing works
Towards Efficient Generative Large Language Model Serving: A Survey from Algorithms to Systems
Miao, X., Oliaro, G., Zhang, Z., Cheng, X., Jin, H., Chen, T., & Jia, Z. (2025) · ACM Computing Surveys · doi:10.1145/3754448 · Cited by 107 · see citing works
LLM Inference Serving: Survey of Recent Advances and Opportunities
Li, B., Jiang, Y., Gadepally, V. N., & Tiwari, D. (2024) · IEEE High Performance Extreme Computing Conference · doi:10.1109/hpec62836.2024.10938426 · Cited by 42 · see citing works
SOFA: A Compute-Memory Optimized Sparsity Accelerator via Cross-Stage Coordinated Tiling
Wang, H., Fang, J., Tang, X., Yue, Z., Li, J., Qin, Y., Guan, S., Yang, Q., Wang, Y., Li, C., et al. (2024) · IEEE/ACM International Symposium on Microarchitecture (MICRO) · doi:10.1109/micro61859.2024.00093 · Cited by 34 · see citing works
Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving
Qin, R., Li, Z., He, W., Zhang, M., Wu, Y., Zheng, W., & Xu, X. (2024) · arXiv · doi:10.48550/arxiv.2407.00079 · Cited by 34 · see citing works
Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving
Qin, R., Li, Z., He, W., Cui, J., Tang, H., Ren, F., Ma, T., Cai, S., Zhang, Y., Zhang, M., et al. (2026) · ACM Transactions on Storage · doi:10.1145/3773772 · Cited by 34 · see citing works
POD-A <scp>ttention</scp> : Unlocking Full Prefill-Decode Overlap for Faster LLM Inference
Kamath, A. K., Prabhu, R., Mohan, J., Peter, S., Ramjee, R., & Panwar, A. (2025) · Proceedings of the 30th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2 · doi:10.1145/3676641.3715996 · Cited by 26 · see citing works
Zhang, G., Lu, C., & Luo, Q. (2025) · Buildings · doi:10.3390/buildings15111944 · Cited by 17 · see citing works
An I/O Characterizing Study of Offloading LLM Models and KV Caches to NVMe SSD
Ren, Z., Doekemeijer, K., Matteis, T. D., Pinto, C., Stoica, R., & Trivedi, A. (2025) · Workshop on Challenges and Opportunities of Efficient and Performant Storage Systems (CHEOPS) · doi:10.1145/3719330.3721230 · Cited by 12 · see citing works
SCOPE : Performance Testing for Serverless Computing
Wen, J., Chen, Z., Zhao, J., Sarro, F., Ping, H., Zhang, Y., Wang, S., & Liu, X. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3717609 · Cited by 9 · see citing works
Self-citationTightLLM: Maximizing Throughput for LLM Inference via Adaptive Offloading Policy
Hu, Y., Liu, X., Yang, G., Li, L., Zeng, K., Zhao, Z., Chen, S., Zhao, L., Li, W., & Li, K. (2025) · IEEE Transactions on Computers · doi:10.1109/tc.2025.3558009 · Cited by 8 · see citing works
Self-citationDeploying Foundation Model Powered Agent Services: A Survey
Xu, W., Chen, J., Zheng, P., Yi, X., Tian, T., Zhu, W., Wan, Q., Wang, H., Fan, Y., Su, Q., et al. (2025) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2025.3580745 · Cited by 16 · see citing works
MCaM : Efficient LLM Inference with Multi-tier KV Cache Management
Chu, K., Shen, Z., Cheng, S.-R., Xiang, D., Liu, Z., & Zhang, W. (2025) · IEEE International Conference on Distributed Computing Systems (ICDCS) · doi:10.1109/icdcs63083.2025.00062 · Cited by 7 · see citing works
ELLIE: Energy-Efficient LLM Inference at the Edge Via Prefill-Decode Splitting
Fan, H., Lin, Y.-C., & Prasanna, V. K. (2025) · IEEE International Conference on Application-specific Systems, Architectures and Processors (ASAP) · doi:10.1109/asap65064.2025.00031 · Cited by 6 · see citing works
SAGESERVE: Optimizing LLM Serving on Cloud Data Centers with Forecast Aware Auto-Scaling
Jaiswal, S., Jain, K., Simmhan, Y. L., Parayil, A., Mallick, A., Wang, R., Amant, R. S., Bansal, C., Rühle, V., Kulkarni, A., et al. (2025) · Proceedings of the ACM on Measurement and Analysis of Computing Systems · doi:10.1145/3771576 · Cited by 14 · see citing works
Liu, M., Gu, J., & Zhao, T. (2025) · IEEE International Conference on Web Services (ICWS) · doi:10.1109/icws67624.2025.00045 · Cited by 5 · see citing works
Characterizing the Behavior and Impact of KV Caching on Transformer Inferences Under Concurrency
Ye, J., Cernuda, J., Maurya, A., Sun, X., Kougkas, A., & Nicolae, B. (2025) · IEEE International Parallel and Distributed Processing Symposium (IPDPS) · doi:10.1109/ipdps64566.2025.00108 · Cited by 4 · see citing works
BucketServe: Bucket-Based Dynamic Batching for Smart and Efficient LLM Inference Serving
Zheng, W., Xu, M., Song, S. H., & Ye, K. (2025) · IEEE International Conference on Internet of Things (iThings) and IEEE Green Computing and Communications (GreenCom) and IEEE Cyber, Physical and Social Computing (CPSCom) and IEEE Smart Data (SmartData) · doi:10.1109/ithings-greencom-cpscom-smartdata-cybermatics67059.2025.00041 · Cited by 3 · see citing works
Zhang, C., Zhu, X., Chen, L., Yang, T., Pan, E., Yu, G., Zhao, Y., Wu, X., Li, B., Mao, W., et al. (2025) · Integrated Circuits and Systems · doi:10.23919/ics.2025.3568404 · Cited by 3 · see citing works
WhisperFlow: speech foundation models in real time
Wang, R., Xu, Z., & Lin, F. X. (2025) · International Conference on Mobile Systems, Applications, and Services · doi:10.1145/3711875.3729151 · Cited by 3 · see citing works
Tropical: Enhancing SLO Attainment in Disaggregated LLM Serving via SLO-Aware Multiplexing
Ma, J., Chen, J., Li, X., Duan, J., Duanmu, H., Zhang, X., Yang, C., & Lin, D. (2025) · ACM/IEEE Design Automation Conference (DAC) · doi:10.1109/dac63849.2025.11132617 · Cited by 2 · see citing works
Liakopoulos, D., Sinha, P., Hu, T., Lee, M., & Yadwadkar, N. J. (2025) · International Conference for High Performance Computing, Networking, Storage and Analysis (SC) · doi:10.1145/3712285.3759867 · Cited by 2 · see citing works
BlitzScale: Fast and Live Large Model Autoscaling with O(1) Host Caching
Zhang, D., Liu, Y., Wang, H., Wei, X., Shan, Y., Chen, R., & Chen, H. Z. (2026) · ACM Transactions on Computer Systems · doi:10.1145/3807606 · Cited by 2 · see citing works
Wu, S., Lin, Y., Peng, S., Chen, W., Ma, C., Shen, M., Chen, L., Xu, C., & Ye, K. (2025) · IEEE International Conference on Cluster Computing (CLUSTER) · doi:10.1109/cluster59342.2025.11186463 · Cited by 1 · see citing works
He, Y., Xu, M., Wu, J., Hu, J., Ma, C., Shen, M. X., Chen, L., Xu, C., Qu, L., & Ke-jiang, Y. (2026) · Software Practice and Experience · doi:10.1002/spe.70054 · Cited by 1 · see citing works
Hermes: Efficient Serving of LLM Applications with Probabilistic Demand Modeling
Liu, Y., Gan, Z., Gan, Z., Wang, W., Chen, C., Shan, Y., Chen, X., Han, Z., Zhu, Y., Sun, S., et al. (2026) · ACM Transactions on Architecture and Code Optimization · doi:10.1145/3803390 · Cited by 1 · see citing works
GPT-5 vs Other LLMs in Long Short-Context Performance
Esmi, N., Nezhad-Moghaddam, M., Borhani, F., Shahbahrami, A., Daemdoost, A., & Gaydadjiev, G. (2025) · 2025 3rd International Conference on Foundation and Large Language Models (FLLM) · doi:10.1109/fllm67465.2025.11391194 · Cited by 1 · see citing works
FastTTS: Accelerating Test-Time Scaling for Edge LLM Reasoning
Chen, H., Zhi-wen, M., Lu, G., Liang, S., Ma, L., Luk, W. W., & Fan, H. (2026) · International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) · doi:10.1145/3779212.3790161 · Cited by 1 · see citing works
Towards High-Goodput LLM Serving with Prefill-decode Multiplexing
Chen, Y., Cui, W., Zhao, H., Xu, Z. S., Fan, X., Chen, X., Zhou, Y., Sun, S., He, B., & Chen, Q. (2026) · International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) · doi:10.1145/3779212.3790236 · Cited by 8 · see citing works
TPipe: Efficient Spiking Transformer Training with Time Parallelism and Asynchronous Pipeline
Bao, Y., Lu, Z., Duan, Q., Lv, C., Du, X., Deng, Z., Feng, J., Liu, S., Chen, Y., & Wang, X. J. (2026) · IEEE Conference on Computer Communications (INFOCOM) · doi:10.1109/infocom59046.2026.11571418 · Cited by 1 · see citing works
Self-citationPKAS: Predictive KVCache-Aware Scheduling for Faster LLM and Transformer Inferences
Ye, J., Maurya, A., Chitty-Venkata, K. T., Nicolae, B., Kougkas, A., & Sun, X. (2026) · ACM International Symposium on High-Performance Parallel and Distributed Computing (HPDC) · doi:10.1145/3806645.3807819 · Cited by 1 · see citing works
Aruna, K. B., Kaliraj, V., Sudha, I., & Sureka, V. (2026) · EAI Endorsed Transactions on Internet of Things · doi:10.4108/eetiot.13349
A Performance Study of LLM Inference on Mooncake-Based Clusters
Yoon, J., Ahn, H., Lee, S., Kim, K., Lee, H., Park, J., & Ahn, S. (2025) · IEEE International Conference on Consumer Electronics-Asia (ICCE-Asia) · doi:10.1109/icce-asia67487.2025.11263536
FlexSpark: Robust and Efficient Multi-Device Collaborative Inference over Wireless Network
Shao, Y., Li, H., Hu, S., Du, X., Wu, H., Zhou, J., & Tan, K. (2025) · Asia-Pacific Workshop on Networking (APNet) · doi:10.1145/3735358.3735368
MP3: Mixed-Precision Pipeline Parallelism Framework for Heterogeneous Edge Devices
Xiong, Y., Liu, W., Li, B., Cao, Q., Zhu, Z., Geng, J., Zhang, R., Zu, Y., & Zhou, X. (2026) · IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems · doi:10.1109/tcad.2026.3663279
Towards Resource-Efficient Serverless LLM Inference with SLINFER
Xu, C., Zijun, L. I., Chen, Q., Zhao, H., Tang, X., & Guo, M. (2026) · IEEE International Symposium on High-Performance Computer Architecture (HPCA) · doi:10.1109/hpca68181.2026.11408548
BLADE: Boosting LLM Decoding’s Communication Efficiency in DRAM-based PIM
Zhao, Y., Liu, F., Wang, Z., Li, M., Zhang, M., Chen, C., & Jiang, L. (2026) · Asia and South Pacific Design Automation Conference (ASP-DAC) · doi:10.1109/asp-dac66049.2026.11420462
Zheng, L., Hu, Y., Shen, C., Zhao, Z., Shi, B., Zhang, Y., Zhao, L., Li, W., & Li, K. (2026) · IEEE Transactions on Parallel and Distributed Systems · doi:10.1109/tpds.2026.3683867
An Empirical Study on Latency Reduction Techniques for Large Language Models
Kumar, A. S., Janarthanan, V., Sharma, S., & Palani, K. (2025) · Conference on AI, Science, Engineering, and Technology (AIxSET) · doi:10.1109/aixset65682.2025.00033
Fu, Y., Qin, P. D., & Jiang, C. (2026) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2026.3690731
Cache-Aware Generative NLP: Reducing Latency in Large-Scale Multilingual Systems
Anurag, A. A., Md.Nasif, Islam, S., Hamim, S. A., & Mridha, M. F. (2026) · IEEE International Conference on Quantum Photonics, Artificial Intelligence & Networking (QPAIN) · doi:10.1109/qpain69676.2026.11546324
PEACE: Preemptive and Efficient Cluster Scheduling for LLM Inference with Mixed Prompts
Zhang, Z., & Shen, H. (2026) · IEEE International Parallel and Distributed Processing Symposium (IPDPS) · doi:10.1109/ipdps65963.2026.00105
FLYING SERVING: On-the-Fly Parallelism Switching for Large Language Model Serving
Gao, S., Yin, J., Wang, F., & Dong, W. (2026) · International Conference on Supercomputing · doi:10.1145/3797905.3800525
Runtime-Controlled Evaluation of Speculative Decoding Methods
Kim, S., Kim, J., Cho, Y., & Seo, J. (2026) · Electronics · doi:10.3390/electronics15143179
Tetris: Efficient Long-context LLM Serving with Chunkwise Dynamic Sequence Parallelism
Li, C., Yang, Y., Zheng, X., Yang, Q., Guan, Y., Zheng, S., Chang, L., Liu, S., Liu, X., & Sun, G. (2026) · Proceedings - International Symposium on Computer Architecture/Proceedings · doi:10.1109/isca66397.2026.00098
Self-citationConServe: Contiguity-Preserving Memory Management for Multi-Turn LLM Serving
Li, B. (2026) · Proceedings - International Symposium on Computer Architecture/Proceedings · doi:10.1109/isca66397.2026.00075
Meridian: In-Memory Acceleration for RAG with Document Attention Decomposition
Liu, C., Huang, Y., Liu, H., Zhang, Y., Qihang, Q. I. U., Li, X., Zheng, L., Liao, X., Jin, H., & Xue, J. (2026) · Proceedings - International Symposium on Computer Architecture/Proceedings · doi:10.1109/isca66397.2026.00041
KVServe: Service-Aware KV Cache Compression for Communication-Efficient Disaggregated LLM Serving
Liu, Z., Ma, X., Luo, D., Zhao, H., Lu, B., Huang, W., Gu, Y., Liu, X., Wei, Z., Liu, J., et al. (2026) · Proceedings of the ACM SIGCOMM 2026 Conference · doi:10.1145/3789240.3829139
Self-citationEfficient Remote KV Cache Reuse with GPU-native Video Codec
Mi, L., Wang, W., Chen, J., Cao, T., Dai, H., & Liu, Y. (2026) · ACM SIGCOMM Conference (SIGCOMM) · doi:10.1145/3789240.3829120
Turbo: Efficiently Serving Long-Context Large Language Models with In-Network Aggregation
Wan, Y., Xu, Y., Zhang, C., Huang, Y., Feng, Y., Xu, W. L., Li, J., Xu, M., Wu, W., & Miao, C. (2026) · ACM SIGCOMM Conference (SIGCOMM) · doi:10.1145/3789240.3829140
He, Y., Shi, D., Dai, Y., Chen, D., Fu, H., Tsu, C., Feng, W., & Fu, S. (2026) · International Conference on Intelligent Multimedia, Networking, and Security (IMNS) · doi:10.1109/imns67862.2026.11655252
Managing KV Cache for Coordinated Waiting and Execution Time in LLM Serving
Shen, H., Sen, T., & Tanaka, M. (2026) · International Conference on Computer Communications and Networks (ICCCN) · doi:10.1109/icccn69946.2026.11662646
A Task-Aware Model Size and Precision Selection Framework for LLM Inference in Edge Computing
Yuan, Y., Lu, B., Zhang, S., & Liu, S. (2026) · IEEE International Conference on Smart Internet of Things (SmartIoT) · doi:10.1109/smartiot70864.2026.00023
Self-citationPreServe: Intelligent Management for LMaaS Systems via Hierarchical Prediction
Jiang, Z., Huang, Y., Yu, G., Huang, J., Gu, J., & Lyu, M. R.-T. (2026) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3744916.3773173
Heterogeneous SLO Guaranteed Multi-Resource-Aware Batching in LLM Serving
Shen, H., Sen, T., & He, Y. (2026) · Proceedings of the International Conference on Parallel Processing · doi:10.1145/3832810.3832922
Batched in Back: Characterizing and Optimizing Offline LLM Inference in Production with ACDC
Yang, L., Li, X., Qian, K., Xu, E., Han, M., Zhu, H., He, T., Yin, Z., Zhai, E., Yu, W., et al. (2026) · ACM SIGOPS Symposium on Operating Systems Principles (SOSP) · doi:10.1145/3830418.3843877
Li, H., Lin, S., Fu, F., Zhou, Y., Ji, X., Zhao, Y., Wang, L., Jiang, J., & Cui, B. (2026) · Proceedings of the ACM on Management of Data · doi:10.1145/3837125
D i F low : A System for Micro-Serving Text-to-image Di ffusion Work flows
Yang, L., Li, S.-Q., Feng, T., Jiang, X., Di, Z., Lu, W., Liu, K., Yu, Y., Lan, T., Yang, G., et al. (2026) · ACM SIGOPS Symposium on Operating Systems Principles (SOSP) · doi:10.1145/3830418.3843880
MeshRT: Compile-Time Governed Wafer-Scale Runtime for Low-Latency High-Throughput Inference
He, C., Xu, L., Lu, Z., Huang, Y., Xiao, H., Deng, C., Ma, L., Miao, Z., Yang, F., & Mai, L. (2026) · ACM SIGOPS Symposium on Operating Systems Principles (SOSP) · doi:10.1145/3830418.3843856
LMTracer: Fine-Grained and Real-Time Performance Profiling for Production LLM Systems
Liu, W., He, Y., Zhao, B., Wang, H., Li, Z., & Zhao, J. (2026) · ACM SIGOPS Symposium on Operating Systems Principles (SOSP) · doi:10.1145/3830418.3843871
Janus: Multi-LLM Serving at Production Scale
Zhou, T., Wang, Y., Zhou, Y., Liu, Z., Wang, Z., Peng, Y., Wang, Y., Zhang, Y., Yin, J., Tian, K., et al. (2026) · ACM SIGOPS Symposium on Operating Systems Principles (SOSP) · doi:10.1145/3830418.3843867
ALPS: An Application-Aware, Length-Predictive Scheduler for Efficient LLM Inference at Edge
Chen, C., Jiao, L., & Mortier, R. (2026) · IEEE Transactions on Parallel and Distributed Systems · doi:10.1109/tpds.2026.3741399
Taichi: Elastic Multi-LLM Serving with Coordinated Memory–Compute Multiplexing
Lv, C., Shi, X., Lei, Z., Li, X., Li, X., Qingwen, L., Tang, Z., & Zhao, X. (2026) · IEEE International Conference on Cluster Computing (CLUSTER) · doi:10.1109/cluster68337.2026.00048
- Authors
- Wu, B., Liu, S., Zhong, Y., Sun, P., Liu, X., & Jin, X.
- Affiliations
- Peking University (CN)
- Shanghai Artificial Intelligence Laboratory (CN)
- Venue
- Proceedings of the ACM SIGOPS 30th Symposium on Operating Systems Principles
- Publisher
- ACM
- Published
- 2024-11-04
- Pages
- 640-654
- OpenAlex
- W4404401017
- Cited by
- 62
- References
- 6
- Citations by year
- 2026: 35 · 2025: 24 · 2024: 3
- FWCI
- 18.87
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- ACM SIGOPS Symposium on Operating Systems Principles (SOSP) (publishedVersion)
- arXiv (Cornell University) (submittedVersion, cc-by-sa)
- arXiv (Cornell University)
- Funders
- National Key Research and Development Program of China (2022YFB4500700)
- National Natural Science Foundation of China (62172008, 62325201)
- Topics
- Physical Sciences › Computer Science › Hardware and Architecture › Parallel Computing and Optimization Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- large language model inference, long-context inference, context window, GPU memory efficiency, communication efficiency
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
