Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving

Generative Agents: Interactive Simulacra of Human Behavior

Park, J. S., O'Brien, J., Cai, C. J., Morris, M. R., Liang, P., & Bernstein, M. S. (2023) · Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology · doi:10.1145/3586183.3606763 · Cited by 2,187 · see citing works

Efficient Memory Management for Large Language Model Serving with PagedAttention

Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez, J., Zhang, H., & Stoica, I. (2023) · Proceedings of the 29th Symposium on Operating Systems Principles · doi:10.1145/3600006.3613165 · Cited by 1,817 · see citing works

DeepSpeed- Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scale

Aminabadi, R. Y., Rajbhandari, S., Awan, A. A., Li, C., Li, D., Zheng, E., Ruwase, O., Smith, S., Zhang, M., Rasley, J., et al. (2022) · SC22: International Conference for High Performance Computing, Networking, Storage and Analysis · doi:10.1109/sc41404.2022.00051 · Cited by 297 · see citing works

CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Serving

Liu, Y., Li, H., Cheng, Y., Ray, S., Huang, Y., Zhang, Q., Du, K., Yao, J., Lu, S., Ananthanarayanan, G., et al. (2024) · Proceedings of the ACM SIGCOMM 2024 Conference · doi:10.1145/3651890.3672274 · Cited by 118 · see citing works

LoongServe: Efficiently Serving Long-Context Large Language Models with Elastic Sequence Parallelism

Wu, B., Liu, S., Zhong, Y., Sun, P., Liu, X., & Jin, X. (2024) · Proceedings of the ACM SIGOPS 30th Symposium on Operating Systems Principles · doi:10.1145/3694715.3695948 · Cited by 62 · see citing works

System Optimizations for Enabling Training of Extreme Long Sequence Transformer Models

Jacobs, S. A., Tanaka, M., Zhang, C., Zhang, M., Aminabadi, R. Y., Song, S. L., Rajbhandari, S., & He, Y. (2024) · IEEE International Symposium on Parallel & Distributed Processing, Workshops and Phd Forum · doi:10.1109/ipdpsw63119.2024.00208 · Cited by 17 · see citing works

Reducing Transformer Key-Value Cache Size with Cross-Layer Attention

Brandon, W., Mishra, M., Nrusimha, A., Panda, R., & Ragan‐Kelley, J. (2024) · neural information processing systems · doi:10.52202/079017-2758 · Cited by 17 · see citing works

Medusa: Accelerating Serverless LLM Inference with Materialization

Zeng, S., Xie, M., Gao, S. Y., Chen, Y., & Lu, Y. (2025) · International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) · doi:10.1145/3669940.3707285 · Cited by 13 · see citing works

Past-Future Scheduler for LLM Serving under SLA Guarantees

Gong, R., Bai, S., Wu, S., Fan, Y., Wang, Z., Li, X., Yang, H., & Liu, X. (2025) · International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) · doi:10.1145/3676641.3716011 · Cited by 15 · see citing works

LLM Knowledge-Driven Target Prototype Learning for Few-Shot Segmentation

Li, P., Liu, F., Jiao, L., Li, S., Liu, X., Chen, P., Li, L., & Hao, Z. (2025) · Knowledge-Based Systems · doi:10.1016/j.knosys.2025.113149 · Cited by 12 · see citing works

An I/O Characterizing Study of Offloading LLM Models and KV Caches to NVMe SSD

Ren, Z., Doekemeijer, K., Matteis, T. D., Pinto, C., Stoica, R., & Trivedi, A. (2025) · Workshop on Challenges and Opportunities of Efficient and Performant Storage Systems (CHEOPS) · doi:10.1145/3719330.3721230 · Cited by 12 · see citing works

Trending Applications of Large Language Models: A User Perspective Survey

Zhang, Y., Zhao, M., Zhang, Y., & Cheung, Y. (2025) · IEEE Transactions on Artificial Intelligence · doi:10.1109/tai.2025.3620272 · Cited by 12 · see citing works

METIS: Fast Quality-Aware RAG Systems with Configuration Adaptation

Ray, S., Pan, R., Gu, Z., Du, K., Feng, S., Ananthanarayanan, G., Netravali, R., & Jiang, J. (2025) · ACM SIGOPS Symposium on Operating Systems Principles (SOSP) · doi:10.1145/3731569.3764855 · Cited by 17 · see citing works

ByteDance Jakiro: Enabling RDMA and TCP over Virtual Private Cloud

Liu, Y., Jiang, L., Li, D., Kang, D., Wei, Z., Chai, Y., Niu, B., Lin, K. L., Ding, X., Pi, J., et al. (2025) · ACM SIGCOMM Conference (SIGCOMM) · doi:10.1145/3718958.3750496 · Cited by 6 · see citing works

AugFL: Augmenting Federated Learning With Pretrained Models

Yue, S., Qin, Z., Deng, Y., Ren, J., Zhang, Y., & Zhang, J. (2025) · IEEE Transactions on Networking · doi:10.1109/ton.2025.3548277 · Cited by 5 · see citing works

Data+AI: LLM4Data and Data4LLM

Li, G., Wang, J., Zhang, C., & Wang, J. (2025) · ACM SIGMOD International Conference on Management of Data (SIGMOD) · doi:10.1145/3722212.3725641 · Cited by 5 · see citing works

Identification of Syndrome Types in Patients With Pancreatic Cancer From Free Text in Electronic Medical Records: Model Development and Validation

Ba, H., Du, H., Cheng, C., Zhang, Y., Ruan, L., & Chen, Z. (2025) · JMIR Formative Research · doi:10.2196/70602 · Cited by 3 · see citing works

From Human Memory to AI Memory: A Survey on Memory Mechanisms in the Era of LLMs

Wu, Y., Liang, S., Zhang, C., Wang, Y., Zhang, Y., Guo, H., Tang, R., & Liu, Y. (2025) · arXiv · doi:10.48550/arxiv.2504.15965 · Cited by 3 · see citing works

Preble: Efficient Distributed Prompt Scheduling for LLM Serving

Srivatsa, V., He, Z., Abhyankar, R., Li, D., & Zhang, Y. (2024) · arXiv · doi:10.48550/arxiv.2407.00023 · Cited by 1 · see citing works

Compactor: Calibrated Query-Agnostic KV Cache Compression with Approximate Leverage Scores

Chari, V., & Van Durme, B. (2025) · arXiv · doi:10.48550/arxiv.2507.08143 · Cited by 1 · see citing works

End‐to‐Network Load Balancing for AI Data Center Networks: A Convergence‐Based Approach to Enhance Training Efficiency

Zhang, R., Zhao, X., Han, Y., Yang, Y., Ruan, J., Zhang, J., Chen, D., & Wang, H. (2025) · Transactions on Emerging Telecommunications Technologies · doi:10.1002/ett.70249 · Cited by 1 · see citing works

Kimi K3: Open Frontier Intelligence

Kimi Team, Bai, T., Bai, Y., Bao, Y., C., M., Cai, J., Cai, X., Cao, P., Cao, Y., Chai, Z., et al. (2026) · arXiv · doi:10.48550/arxiv.2607.24653

FlowKV: A Disaggregated Inference Framework with Low-Latency KV Cache Transfer and Load-Aware Scheduling

Li, W., Jiang, G., Ding, X., Tao, Z., Hao, C., Xu, C., Zhang, Y., & Wang, H. (2025) · arXiv · doi:10.48550/arxiv.2504.03775

CritiPrefill: A Segment-wise Criticality-based Approach for Prefilling Acceleration in LLMs

Lv, J., Feng, Y., Xie, X., Jia, X., Peng, Q., & Xie, G. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10887916

Performance Analysis of Chinese Large Language Models in Solving Math Word Problems

Pan, S., Liu, K., Chen, W., & He, B. (2024) · 2024 International Conference on Intelligent Education and Intelligent Research (IEIR) · doi:10.1109/ieir62538.2024.10960109 · Cited by 1 · see citing works

Design Considerations for LLM Inference in Data Centers: Chip and Interconnect

Shi, Z., Wang, Y., Diao, M., Wu, W., Su, Z., Li, X., Niu, R., Zang, H., Song, K., Yu, Y., et al. (2025) · IEEE Symposium on VLSI Technology and Circuits (VLSI Technology and Circuits) · doi:10.23919/vlsitechnologyandcir65189.2025.11074875

Libra: Novel LLM Token Streaming via Region-Based Task Scheduling and Token Bundling

Zhou, C., Liu, S., Yuan, X., Shi, J., Zhang, Y., & Pu, L. (2025) · International Workshop on Quality of Service · doi:10.1109/iwqos65803.2025.11143346

JUNO++: Optimizing ANNS and Enabling Efficient Sparse Attention in LLM via Ray Tracing Core

Liu, Z., Ni, W., Leng, J., Feng, Y., Guo, C., Chen, Q., Li, C., Guo, M., Ma, Y., Zhang, F., et al. (2025) · ACM Transactions on Architecture and Code Optimization · doi:10.1145/3768585

AsyncGrid: An Intralayer and Interlayer Asynchronous Hybrid Parallelism System for Responsive Edge LLM Inference

Xiong, Y., Zhang, R., Zu, Y., Liu, W., Zhu, Z., Geng, J., Li, B., Cao, Q., & Zhou, X. (2025) · IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems · doi:10.1109/tcad.2025.3624135

TokenFlow: Token-Level GPU Sharing and Adaptive Scheduling for Multi-Model Concurrent LLM Inference

Zeng, K., Huang, Z., Yang, Y., Zhang, X., Huang, S.-Y., & Meng, R. (2026) · International Conference on Electronic Technology and Information Science (ICETIS) · doi:10.1109/icetis70504.2026.11633493

ARION: Aggregated Routing for In-Order Optimized Network Load Balancing in Data Centers

Korenfeld, E., Kampeas, J., & Gurewitz, O. (2026) · IEEE Transactions on Networking · doi:10.1109/ton.2026.3678315

LLM-Prefetch: Accelerating Long-Context LLM Inference via Explicit Prefetching

Wang, S. (2026) · International Symposium on Computer Applications and Information Systems (ISCAIS) · doi:10.1109/iscait69154.2026.11477382

PiLLM: Resource-Efficient LLM Inference Using Workload Prediction

Fan, Y., Bai, S., Gong, R., Wang, Z., & Fan, R. (2026) · European Conference on Computer Systems · doi:10.1145/3767295.3769393

FinePruner: Unbiased Attention-Head-Level Fine-Grained Token Reduction for Efficient Inference of Large Vision-Language Models

Wang, Z., Zheng, X., & Peng, Y. (2026) · IEEE Transactions on Image Processing · doi:10.1109/tip.2026.3687073

Redefining AI Architecture with NAND Flash

Lee, H.-Z., Wang, M.-H., Wang, Y.-H., Hsu, C.-M., Du, Y.-J., Li, A.-C., Chen, T.-Y., Tsou, K.-W., Tsao, S.-M., Syu, J.-P., et al. (2026) · International Symposium on VLSI Technology, Systems and Applications (VLSI-TSA) · doi:10.1109/vlsitsa69131.2026.11528145

Model Splitting and Computing Resource Allocation for Collaborative Edge-Device LLM Inference: A Transformer-Enhanced DRL Approach

Chen, X., Guo, F., Liu, C., Peng, M., & Quek, T. Q. S. (2026) · IEEE Wireless Communications and Networking Conference · doi:10.1109/wcnc65185.2026.11555072

Dynamo-MoE: Accelerating Sparse Large Model Inference with Dynamic Parallelization

Chen, J.-H., Li, S.-G., Fu, R.-T., Wu, T., Ma, Z., & Dong, J. (2026) · IEEE International Symposium on High-Performance Parallel Distributed Computing · doi:10.1145/3806645.3807820

Kairos: SLO-Aware Scheduling and a Benchmark for Multi-Tenant LLM Inference Serving under Bursty, Heterogeneous Load

Gullapalli, S., & Divi, V. R. (2026) · International Conference on Electronics Representation and Algorithm (ICERA) · doi:10.1109/icera72709.2026.11666706

Edge-Assisted KV-Cache Collaborative Loading for Split LLM Inference

Zhang, S., Xie, Y., Sun, L., Wu, H., & Zhang, L. (2026) · Lecture notes in computer science · doi:10.1007/978-981-92-3400-4_16

Path-Compressed Trie for Efficient KV Cache Reuse

Wang, B., & Xu, Y. (2026) · International Conference on Computer and Communication Systems (ICCCS) · doi:10.1109/icccs69761.2026.11613300

Prototype-based federated learning for one-time soft clustering

Xie, P., Gao, X., Wu, J., Zhang, G., Ma, S., & Li, Y. (2026) · Cluster Computing · doi:10.1007/s10586-026-06431-z

Efficient VLM Inference System With LoRA Adapters at the Edge

Wang, W., Mi, L., Chen, J., Yu, T., Cao, T., Li, Y., Li, M., Dai, H., Chen, G., & Liu, Y. (2026) · IEEE Transactions on Mobile Computing · doi:10.1109/tmc.2026.3723365

EC-RAG: Towards Efficient Edge-Cloud Retrieval-Augmented Generation Systems

Wang, L., Wang, K., Jia, R., Lu, K., Wan, J., Huo, H., Zhai, Y., Liang, Z., & Wang, D. (2026) · IEEE International Conference on Data Engineering (ICDE) · doi:10.1109/icde65706.2026.00082

ParaCascade: A Parallel Cascading Framework Supporting Early Routing

Wei, H., Yin, L., Chen, C., Sun, J., Zhang, M., & Peng, Y. (2026) · IEEE International Conference on Joint Cloud Computing (JCC) · doi:10.1109/jcc72984.2026.00031

FlowCache: Workflow-Aware Intermediate-State Reuse for Multi-Tool LLM Serving

Wang, F., Li, M., Sheng, F., & Yang, K. (2026) · International Conference on Computer Science and Blockchain (CCSB) · doi:10.1109/ccsb71176.2026.11716474 · Cited by 1 · see citing works

EcoxPU: Autonomous Multi-Agent Orchestration for Energy-Efficient Disaggregated LLM Inference on Heterogeneous xPUs

Dong, D., Wu, C., Chen, Z., Yan, L., Niu, J., Yu, J., Ma, Z., & Zhai, J. (2026) · IEEE Transactions on Parallel and Distributed Systems · doi:10.1109/tpds.2026.3732699

Predictive Multi-Tier Memory Management for KV Cache in Large-Scale GPU Inference

Ganjihal, S. R. (2026) · arXiv · doi:10.48550/arxiv.2604.26968

A Performance Study of LLM Inference on Mooncake-Based Clusters

Yoon, J., Ahn, H., Lee, S., Kim, K., Lee, H., Park, J., & Ahn, S. (2025) · IEEE International Conference on Consumer Electronics-Asia (ICCE-Asia) · doi:10.1109/icce-asia67487.2025.11263536

MCaM : Efficient LLM Inference with Multi-tier KV Cache Management

Chu, K., Shen, Z., Cheng, S.-R., Xiang, D., Liu, Z., & Zhang, W. (2025) · IEEE International Conference on Distributed Computing Systems (ICDCS) · doi:10.1109/icdcs63083.2025.00062 · Cited by 7 · see citing works

Towards Efficient Large Language Model Serving: A Survey on System-Aware KV Cache Optimization

Jiang, J., Yang, P., Zhang, R., & Liu, F. (2026) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2026.findings-acl.1916 · Cited by 5 · see citing works

Multi-Region Cloud-Native LLM Inference and Dynamic KV-Cache RoutingPreprint

Singh, S. K. (2026) · Zenodo (CERN European Organization for Nuclear Research) · doi:10.5281/zenodo.23513188

KVCache Cache in the Wild: Characterizing and Optimizing KVCache Cache at a Large Cloud ProviderPreprint

Wang, J., Han, J., Wei, X., Shen, S., Zhang, D., Fang, C., Chen, R., Yu, W., & Chen, H. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2506.02634 · Cited by 3 · see citing works