The DOI matches its Crossref record; 39 of the 39 references it cites were found in a registry or an index.
RAGCache: Efficient Knowledge Caching for Retrieval-Augmented Generation
Open accessAbstract
Retrieval-Augmented Generation (RAG) has demonstrated substantial advancements in various natural language processing tasks by integrating the strengths of large language models (LLMs) and external knowledge databases. However, the retrieval step introduces long sequence generation and extra data dependency, resulting in long end-to-end latency. Our analysis benchmarks current RAG systems and reveals that, while the retrieval step poses performance challenges, it also offers optimization opportunities through its retrieval pattern and streaming search behavior. We propose RAGCache, a latency-optimized serving system tailored for RAG. RAGCache leverages the retrieval pattern to organize and cache the intermediate states of retrieved knowledge in a knowledge tree across the GPU and host memory hierarchy, reducing LLM generation time. RAGCache employs dynamic speculative pipelining to exploit the streaming search behavior, overlapping retrieval with LLM generation to minimize end-to-end latency. We implement RAGCache based on vLLM and Faiss, and evaluate it on both open-source and production datasets. Experimental results demonstrate that RAGCache reduces the time to first token (TTFT) by up to 4× and improves the throughput by up to 2.1× compared to vLLM integrated with Faiss.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“previous work on singleLLM KV cache sharing has shown up to 8 × latency and throughput improvement by speeding up the expensive prefill phase of LLM inference, particularly for long context workloads [31,32,49,61].”
From DroidSpeak: KV Cache Sharing for Cross-LLM Communication and Multi-LLM Serving (2024) · cites this paper as [49]
Citations
Cites
PyTorch: An Imperative Style, High-Performance Deep Learning Library
Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., et al. (2019) · arXiv · doi:10.48550/arxiv.1912.01703 · Cited by 31,473
LLaMA: Open and Efficient Foundation Language Models
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. (2023) · arXiv · doi:10.48550/arxiv.2302.13971 · Cited by 4,602
Natural Questions: A benchmark for question answering research
Kwiatkowski, T., Palomaki, J., Redfield, O., Collins, M., Parikh, A., Alberti, C., Epstein, D., Polosukhin, I., Devlin, J., Lee, K., et al. (2019) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00276 · Cited by 2,069
Malkov, Y. A., & Yashunin, D. A. (2020) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2018.2889473 · Cited by 1,904
Efficient Memory Management for Large Language Model Serving with PagedAttention
Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez, J., Zhang, H., & Stoica, I. (2023) · Proceedings of the 29th Symposium on Operating Systems Principles · doi:10.1145/3600006.3613165 · Cited by 1,817
Efficient Memory Management for Large Language Model Serving with PagedAttention
Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez, J. E., Zhang, H., & Stoica, I. (2023) · arXiv · doi:10.48550/arxiv.2309.06180 · Cited by 1,802
TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension
Joshi, M., Choi, E., Weld, D., & Zettlemoyer, L. (2017) · Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p17-1147 · Cited by 1,690 · cited 1×
“Prefill Latency = Miss Rate × Full Prefill Latency + ( 1 − Miss Rate ) × Cache Hit Latency”
TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension
Joshi, M., Choi, E., Weld, D. S., & Zettlemoyer, L. (2017) · arXiv · doi:10.48550/arxiv.1705.03551 · Cited by 1,650 · cited 1×
“Prefill Latency = Miss Rate × Full Prefill Latency + ( 1 − Miss Rate ) × Cache Hit Latency”
Lost in the Middle: How Language Models Use Long Contexts
Liu, N. F., Lin, K., Hewitt, J., Paranjape, A., Bevilacqua, M., Petroni, F., & Liang, P. (2024) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00638 · Cited by 1,612
Lost in the Middle: How Language Models Use Long Contexts
Liu, N. F., Lin, K., Hewitt, J., Paranjape, A., Bevilacqua, M., Petroni, F., & Liang, P. (2023) · arXiv · doi:10.48550/arxiv.2307.03172 · Cited by 1,599
Research works printed but not linked: 36
- Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. 2020. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in Neural Information Processing Systems 33 (2020), 9459–9474.
- Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D. Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. Advances in Neural Information Processing Systems 33 (2020), 1877–1901.
- Lianmin Zheng Liangsheng Yin Zhiqiang Xie Jeff Huang Chuyue Sun Cody Hao Yu Shiyi Cao Christos Kozyrakis Ion Stoica Joseph E. Gonzalez Clark Barrett and Ying Sheng. 2023. Efficiently programming large language models using sglang. arXiv:2312.07104. Retrieved from https://arxiv.org/abs/arXiv:2312.07104 · arXiv
- Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. 2023. Tree of thoughts: Deliberate problem solving with large language models. Advances in Neural Information Processing Systems 36 (2023), 11809–11822.
- Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in Neural Information Processing Systems 30 (2017).
- 10.1109/TPAMI.2014.2361319 · DOI · Paper page
- Wenqi Jiang Shuai Zhang Boran Han Jie Wang Bernie Wang and Tim Kraska. 2024. Piperag: Fast retrieval-augmented generation via algorithm-system co-design. arXiv:2403.05676. Retrieved from https://arxiv.org/abs/arXiv:2403.05676 · arXiv
- Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Millican, George Bm Van Den Driessche, Jean-Baptiste Lespiau, Bogdan Damoc, Aidan Clark, Diego de Las Casas, Aurelia Guy, Jacob Menick, Roman Ring, Tom Hennigan, Saffron Huang, Loren Maggiore, Chris Jones, Albin Cassirer, Andy Brock, Michela Paganini, Geoffrey Irving, Oriol Vinyals, Simon Osindero, Karen Simonyan, Jack W. Rae, Erich Elsen, and Laurent Sifre. 2022. Improving language models by retrieving from trillions of tokens. In Proceedings of the International Conference on Machine Learning. PMLR, 2206–2240.
- Daniel Adiwardana Minh-Thang Luong David R. So Jamie Hall Noah Fiedel Romal Thoppilan Zi Yang Apoorv Kulshreshtha Gaurav Nemade Yifeng Lu and Quoc V. Le. 2020. Towards a human-like open-domain chatbot. arXiv:2001.09977. Retrieved from https://arxiv.org/abs/arXiv:2001.09977 · arXiv
- Shuai Lu Nan Duan Hojae Han Daya Guo Seung-won Hwang and Alexey Svyatkovskiy. 2022. Reacc: A retrieval-augmented code completion framework. arXiv:2203.07722. Retrieved from https://arxiv.org/abs/arXiv:2203.07722 · arXiv
- Zili Zhang, Chao Jin, Linpeng Tang, Xuanzhe Liu, and Xin Jin. 2023. Fast, approximate vector queries on very large unstructured datasets. In Proceedings of the 20th USENIX Symposium on Networked Systems Design and Implementation (NSDI 23). 995–1011.
- Zili Zhang, Fangyue Liu, Gang Huang, Xuanzhe Liu, and Xin Jin. 2024. Fast vector query processing for large datasets beyond \(\lbrace\) GPU \(\rbrace\) memory with reordered pipelining. In Proceedings of the 21st USENIX Symposium on Networked Systems Design and Implementation (NSDI 24). 23–40.
- Dan Hendrycks Collin Burns Steven Basart Andy Zou Mantas Mazeika Dawn Song and Jacob Steinhardt. 2020. Measuring massive multitask language understanding. arXiv:2009.03300. Retrieved from https://arxiv.org/abs/arXiv:2009.03300 · arXiv
- Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W. Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. 2018. HotpotQA: A dataset for diverse, explainable multi-hop question answering. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP).
- Ludmila Cherkasova. 1998. Improving WWW proxies performance with greedy-dual-size-frequency caching policy. Hewlett-Packard Laboratories Palo Alto, CA, USA.
- Gyeong-In Yu, Joo Seong Jeong, Geon-Woo Kim, Soojeong Kim, and Byung-Gon Chun. 2022. Orca: A distributed serving system for \(\lbrace\) Transformer-Based \(\rbrace\) generative models. In Proceedings of the 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22). 521–538.
- Amey Agrawal, Nitin Kedia, Ashish Panwar, Jayashree Mohan, Nipun Kwatra, Bhargav Gulavani, Alexey Tumanov, and Ramachandran Ramjee. 2024. Taming throughput-latency tradeoff in LLM inference with Sarathi-serve. In Proceedings of the 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24). 117–134.
- 10.1109/ISCA59077.2024.00019 · DOI · Paper page
- DeepSeek-AI Aixin Liu Bei Feng Bing Xue Bingxuan Wang Bochao Wu Chengda Lu Chenggang Zhao Chengqi Deng Chenyu Zhang2024. DeepSeek-V3 technical report. arXiv:2412.19437. Retrieved from https://arxiv.org/abs/arXiv:2412.19437 · arXiv
- Ruoyu Qin, Zheming Li, Weiran He, Jialei Cui, Feng Ren, Mingxing Zhang, Yongwei Wu, Weimin Zheng, and Xinran Xu. 2025. Mooncake: Trading more storage for less computation — a KVCache-centric architecture for serving LLM chatbot. In Proceedings of the 23rd USENIX Conference on File and Storage Technologies (FAST 25). 155–170.
- Bingyang Wu, Zili Zhang, Zhihao Bai, Xuanzhe Liu, and Xin Jin. 2023. Transparent \(\lbrace\) GPU \(\rbrace\) sharing in container clouds for deep learning workloads. In Proceedings of the 20th USENIX Symposium on Networked Systems Design and Implementation (NSDI 23). 69–85.
- Joshua Ainslie James Lee-Thorp Michiel de Jong Yury Zemlyanskiy Federico Lebrón and Sumit Sanghai. 2023. Gqa: Training generalized multi-query transformer models from multi-head checkpoints. arXiv:2305.13245. Retrieved from https://arxiv.org/abs/arXiv:2305.13245 · DOI · Paper page
- Albert Q. Jiang Alexandre Sablayrolles Antoine Roux Arthur Mensch Blanche Savary Chris Bamford Devendra Singh Chaplot Diego de las Casas Emma Bou Hanna Florian Bressand Gianna Lengyel Guillaume Bour Guillaume Lample Lélio Renard Lavaud Lucile Saulnier Marie-Anne Lachaux Pierre Stock Sandeep Subramanian Sophia Yang Szymon Antoniak Teven Le Scao Théophile Gervet Thibaut Lavril Thomas Wang Timothée Lacroix and William El Sayed. 2024. Mixtral of experts. arXiv:2401.04088. Retrieved from https://arxiv.org/abs/arXiv:2401.04088 · arXiv
- Bingyang Wu Yinmin Zhong Zili Zhang Gang Huang Xuanzhe Liu and Xin Jin. 2023. Fast distributed inference serving for large language models. arXiv:2305.05920. Retrieved from https://arxiv.org/abs/arXiv:2305.05920 · arXiv
- Zhuohan Li, Lianmin Zheng, Yinmin Zhong, Vincent Liu, Ying Sheng, Xin Jin, Yanping Huang, Zhifeng Chen, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. \(\lbrace\) AlpaServe \(\rbrace\) : Statistical multiplexing with model parallelism for deep learning serving. In Proceedings of the 17th USENIX Symposium on Operating Systems Design and Implementation (OSDI 23). 663–679.
- Charlie Chen Sebastian Borgeaud Geoffrey Irving Jean-Baptiste Lespiau Laurent Sifre and John Jumper. 2023. Accelerating large language model decoding with speculative sampling. arXiv:2302.01318. Retrieved from https://arxiv.org/abs/arXiv:2302.01318 · arXiv
- Zhihao Zhang Alan Zhu Lijie Yang Yihua Xu Lanting Li Phitchaya Mangpo Phothilimthana and Zhihao Jia. 2024. Accelerating retrieval-augmented language model serving with speculation. arXiv:2401.14021. Retrieved from https://arxiv.org/abs/arXiv:2401.14021 · arXiv
- Qi Chen, Bing Zhao, Haidong Wang, Mingqin Li, Chuanjie Liu, Zengzhong Li, Mao Yang, and Jingdong Wang. 2021. Spann: Highly-efficient billion-scale approximate nearest neighborhood search. Advances in Neural Information Processing Systems 34 (2021), 5199–5212.
- 10.14778/3303753.3303754 · DOI · Paper page
- Suhas Jayaram Subramanya, Fnu Devvrit, Harsha Vardhan Simhadri, Ravishankar Krishnawamy, and Rohan Kadekodi. 2019. Diskann: Fast accurate billion-point nearest neighbor search on a single node. Advances in Neural Information Processing Systems 32 (2019).
- Guangxuan Xiao Yuandong Tian Beidi Chen Song Han and Mike Lewis. 2023. Efficient streaming language models with attention sinks. arXiv:2309.17453. Retrieved from https://arxiv.org/abs/arXiv:2309.17453 · arXiv
- Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. 2022. Gpt3. int8 (): 8-bit matrix multiplication for transformers at scale. Advances in Neural Information Processing Systems 35 (2022), 30318–30332.
- Zhuohan Li, Eric Wallace, Sheng Shen, Kevin Lin, Kurt Keutzer, Dan Klein, and Joey Gonzalez. 2020. Train big, then compress: Rethinking model size for efficient training and inference of transformers. In Proceedings of the International Conference on machine learning. PMLR, 5958–5968.
- Suyu Ge Yunan Zhang Liyuan Liu Minjia Zhang Jiawei Han and Jianfeng Gao. 2023. Model tells you what to discard: Adaptive kv cache compression for llms. arXiv:2310.01801. Retrieved from https://arxiv.org/abs/arXiv:2310.01801 · arXiv
- Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher Ré, Clark Barrett, Zhangyang Wang, and Beidi Chen. 2023. H2o: Heavy-hitter oracle for efficient generative inference of large language models. Advances in Neural Information Processing Systems 36 (2023), 34661–34710.
- Chien-Yu Lin Keisuke Kamahori Yiyu Liu Xiaoxiang Shi Madhav Kashyap Yile Gu Rulin Shao Zihao Ye Kan Zhu Stephanie Wang Arvind Krishnamurthy Rohan Kadekodi Luis Ceze and Baris Kasikci. 2025. TeleRAG: Efficient retrieval-augmented generation inference with lookahead retrieval. arXiv:2502.20969. Retrieved from https://arxiv.org/abs/arXiv:2502.20969 · arXiv
Cited by
Retrieval-Augmented Generation for AI-Generated Content: A Survey
Zhao, P., Zhang, H., Yu, Q., Wang, Z., Geng, Y., Fu, F., Yang, L., Zhang, W., Jiang, J., & Cui, B. (2024) · arXiv · doi:10.48550/arxiv.2402.19473 · Cited by 240
A survey on knowledge-oriented retrieval-augmented generation
Cheng, M., Luo, Y., Ouyang, J., Liu, Q., Liu, H., Li, L., Yu, S., Zhang, B., Cao, J., Ma, J., et al. (2025) · arXiv · doi:10.48550/arxiv.2503.10677 · Cited by 140
“Other research also considers not only execution time but also hardware utilization, offering insights into the trade-offs between effectiveness and efficiency[66, 106, 209].”
LLM Inference Serving: Survey of Recent Advances and Opportunities
Li, B., Jiang, Y., Gadepally, V. N., & Tiwari, D. (2024) · IEEE High Performance Extreme Computing Conference · doi:10.1109/hpec62836.2024.10938426 · Cited by 157
“RAGCache [47] caches intermediate states of external knowledge with a knowledge tree to organize and store intermediate states.”
Integration of Generative AI and Mobile Networking: A Comprehensive Survey
Li, J., He, M., Zhou, C., Huang, X., Liu, Z., Zhao, L. Y., Wang, C., & Wu, H. (2025) · IEEE Transactions on Network Science and Engineering · doi:10.1109/tnse.2025.3639401 · Cited by 24
Improving the RAG-based Personalized Discharge Care System by Introducing the Memory Mechanism
Yang, Y., Xu, C., Guo, J., Feng, T., & Ruan, C. (2025) · International Conference on Computer Research and Development · doi:10.1109/iccrd64588.2025.10963086 · Cited by 26
“This hybrid approach allows for more accurate, contextually relevant, and up-to-date responses, which is particularly beneficial in applications such as personalized discharge care where the latest medical guidelines and patient-specific data are crucial [7].”
HotPrefix: Hotness-Aware KV Cache Scheduling for Efficient Prefix Sharing in LLM Inference Systems
Li, Y., Gu, R., Huan, C., Wang, Z., Yao, R., Tian, C., & Chen, G. (2025) · Proceedings of the ACM on Management of Data · doi:10.1145/3749168 · Cited by 17
METIS: Fast Quality-Aware RAG Systems with Configuration Adaptation
Ray, S., Pan, R., Gu, Z., Du, K., Feng, S., Ananthanarayanan, G., Netravali, R., & Jiang, J. (2025) · ACM SIGOPS Symposium on Operating Systems Principles (SOSP) · doi:10.1145/3731569.3764855 · Cited by 53
Accelerating Retrieval Augmented Language Model via PIM and PNM Integration
Jang, J.-W., Oh, J., Kong, Y., Hong, J.-Y., Cho, S.-H., Lee, J., Yang, H., & Yang, J.-S. (2025) · IEEE/ACM International Symposium on Microarchitecture (MICRO) · doi:10.1145/3725843.3756020 · Cited by 9
Arısoy, M. V., Yalçınkaya, M. A., Gürfidan, R., & Arısoy, A. (2025) · Applied Sciences · doi:10.3390/app15041715 · Cited by 5
Hong, G., Ouyang, T., Zhao, K., Zhou, Z., & Chen, X. (2025) · IEEE Real-Time Systems Symposium (RTSS) · doi:10.1109/rtss66672.2025.00022 · Cited by 7
“RAGCache [37] significantly reduces latency and boosts throughput by efficiently caching and reusing intermediate states of retrieved documents through a multi-level dynamic caching mechanism.”
Toward Sustainable Agentic AI Systems: A Survey of Architectures and Methodologies
Lee, Y., & Park, E. (2026) · Sustainable Development · doi:10.1002/sd.70942 · Cited by 13
Ruparel, H., & Patel, T. (2025) · Software Impacts · doi:10.1016/j.simpa.2025.100783 · Cited by 3
Malloy, T., Bernardy, L., Bachyr, O. E., Philippy, F., Samhi, J., Klein, J., & Bissyandé, T. F. (2025) · Electronics · doi:10.3390/electronics14193872 · Cited by 4
From Human Memory to AI Memory: A Survey on Memory Mechanisms in the Era of LLMs
Wu, Y., Liang, S., Zhang, C., Wang, Y., Zhang, Y., Guo, H., Tang, R., & Liu, Y. (2025) · arXiv · doi:10.48550/arxiv.2504.15965 · Cited by 3
“Building on these ideas, RAGCache [131] introduces a multilevel dynamic caching system tailored for Retrieval-Augmented Generation (RAG), which caches intermediate knowledge states, optimizes memory replacement policies based on LLM inference and retrieval patterns, and overlaps retrieval with…”
DroidSpeak: KV Cache Sharing for Cross-LLM Communication and Multi-LLM Serving
Liu, Y., Huang, Y., Yao, J., Feng, S., Gu, Z., Du, K., Li, H., Cheng, Y., Jiang, J., Lu, S., et al. (2024) · arXiv · doi:10.48550/arxiv.2411.02820 · Cited by 2
CARINA: An Efficient CXL-Oriented Embedding Serving System for Recommendation Models
Yin, P., Zhou, Q., Yan, X., Wang, C., Lo, E., Li, C., Lu, L., Fan, H. T., Zhou, W., Yang, M.-C., et al. (2025) · Proceedings of the ACM on Management of Data · doi:10.1145/3725274 · Cited by 2
iKnow: an Intent-Guided Chatbot for Cloud Operations with Retrieval-Augmented Generation
Huang, J., Zhong, Y., Yu, G., Jiang, Z., Yan, M., 栾文飞, Yang, T., Ren, R., & Lyu, M. R.-T. (2025) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase63991.2025.00084 · Cited by 2
“Efficiency is critical for industrial QA systems, as high latency can negatively impact user satisfaction [61], [62] In this RQ, we evaluate the efficiency of iKnow by end-to-end and component-wise latency on all the 2,000 samples.”
VectorLiteRAG: Latency-Aware and Fine-Grained Resource Partitioning for Efficient RAG
Kim, J., & Mahajan, D. (2026) · IEEE International Symposium on High-Performance Computer Architecture (HPCA) · doi:10.1109/hpca68181.2026.11408552 · Cited by 19
“RagCache [15] improves the generation throughput by managing the reuse of KV cache between tenants, focusing on scheduling on the LLM side.”
From Navigation to Intention: Reframing the Web Experience through Goal-Driven Interfaces
Cordioli, L., & Matera, M. (2026) · International World Wide Web Conference (WWW) · doi:10.1145/3774904.3792479 · Cited by 2
Skyline Retrieval meets Set-Cover Chunk Merging: A Cost-Effective RAG-Sketch for Long-Context LLM QA
Zhu, X., Li, H., Zhang, Y., & Chen, L. (2026) · Proceedings of the ACM on Management of Data · doi:10.1145/3802111 · Cited by 5
Chimera-VDB: Mixed-Precision Vector Database with HNSW Index for RAG-LLM
Yamane, N., Zielewski, M. R., Nakamura, T., & Suganuma, T. (2025) · ACM SIGOPS Asia-Pacific Workshop on Systems (APSys) · doi:10.1145/3725783.3764411 · Cited by 2
Caching at Scale: Efficiency and Fairness Analysis in Multi-tenant RAG Systems
Ruparel, H., & Patel, T. (2025) · SN Computer Science · doi:10.1007/s42979-025-04467-3 · Cited by 2
“RAGCache introduced a multi-level cache for RAG that stores intermediate states of the LLM for commonly retrieved documents [2].”
InfluentialFunctional Programming Meets Pinecone: Recommending Graph Structured Documents
Δρακόπουλος, Γ., Theodorakopoulos, L., Sioutas, S., & Phivos Mylonas, B. (2025) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata66926.2025.11402214 · Cited by 1
“…graphs for augmenting financial data [6], supporting large language models (LLMs) for education [7], developing bioinformatics knowledge bases [8], knowledge cache [9], and computing similarity between vectors with attributes pertaining to the MBTI of people extracted from written text [10].”
PKAS: Predictive KVCache-Aware Scheduling for Faster LLM and Transformer Inferences
Ye, J., Maurya, A., Chitty-Venkata, K. T., Nicolae, B., Kougkas, A., & Sun, X. (2026) · ACM International Symposium on High-Performance Parallel and Distributed Computing (HPDC) · doi:10.1145/3806645.3807819 · Cited by 1
Enhancing RAG Systems: A Survey of Optimization Strategies for Performance and Scalability
Nishanth, S., & Swetha, S. (2024) · INTERANTIONAL JOURNAL OF SCIENTIFIC RESEARCH IN ENGINEERING AND MANAGEMENT · doi:10.55041/ijsrem35402 · Cited by 4
PEEK: Predictive Queue-Informed KV Cache Management for LLM Serving
Xie, B., Wang, Z., Tanaka, M., & Zheng, Z. (2026) · arXiv · doi:10.48550/arxiv.2607.02525
“…axis supplies the capability to share KV across prompts: PromptCache [Gim et al., 2024] (module-level KV via positional re-encoding), RAGCache [Jin et al., 2024] (hierarchical RAG cache), ChunkAttention [Ye et al., 2024] (attention over a prefix tree), Hydragen [Juravsky et al., 2024] (fused…”
“A closer analysis reveals that GAIA’s heterogeneous task space, ranging from video dialog reasoning to sales computation, limits the effectiveness of keyword-based cache retrieval, as many task descriptions are highly specific and rarely recur.”
SmoothAgent: Efficient Long-Horizon LLM-Based Agent Serving with Lookahead Context Engineering
Pan, Z., Wang, Q., Hu, Z., Chen, C., Guan, Y., Zhou, Y., Swanson, S., & Ding, Y. (2026) · arXiv · doi:10.48550/arxiv.2607.00151
Discovery Omnia: Dynamic RAG for Enhanced Patent Analysis and Systematic Innovation
Avogadri, S., Alzetta, G., & Russo, D. (2025) · IFIP advances in information and communication technology · doi:10.1007/978-3-032-08847-5_6 · Cited by 1
Accelerating RAG Systems: A Performance-Oriented Systematic Mapping
da Silva, J. G. J., de Oliveira, S. S. T., & Filho, A. R. G. (2025) · doi:10.5753/eniac.2025.12304
“These techniques aim to balance the trade-off between latency reduction and memory footprint, particularly in scenarios with extended interaction histories or high-frequency queries [Jin et al. 2024].”
Park, J., & Han, S. (2025) · Journal of Construction Automation and Robotics · doi:10.55785/jcar.4.4.9
Liu, Y., Wang, Z., Chen, Y., He, Y., Lian, J., & Lu, H. (2025) · IEEE International Conference on Data Science in Cyberspace · doi:10.1109/dsc67331.2025.00010
Bauhaus: Restructuring Vector Database for LLM Retrieval on CXL-Based Tiered Memory
Kim, K., 안성수, Jeong, W., Kim, J., Choi, S., Gil, M., Kim, M., Jung, D., Hong, Y., Jung, H., et al. (2026) · IEEE Transactions on Computers · doi:10.1109/tc.2026.3656215 · Cited by 2
DKC-LLM: Dynamic Knowledge Caching for Large Language Models in Business Applications
Khaliq, A., & Adebayo, K. J. (2026) · IEEE Access · doi:10.1109/access.2026.3662344
“Frameworks such as RagCache [8], PipeRAG [26], CacheCraft [30], and CacheBlend [31] aim to reuse retrieval results or generated responses.”
InfluentialElectricity Standard Knowledge Generation Based on Large Language Model
Fan, X., Zhang, S. Z. F., Liang, X., Wang, Z., & Peng, T. (2025) · International Conference on Power System and Energy Internet (PoSEI) · doi:10.1109/posei67825.2025.11382276
Hybrid Retrieval for Power Equipment Standard Knowledge
Zhang, S. Z. F., Fan, X., Song, B., Liang, X., Zhang, Q., Wang, Z., Xu, X., & Zou, Y. (2025) · International Conference on Smart Grid and Energy Engineering (SGEE) · doi:10.1109/sgee68429.2025.11385482
“Although progress has been made, existing hybrid retrieval methods still have room for improvement, especially considering the heterogeneous multi-source data, complex terminology, and diverse query types characteristic of power technical standards[17].”
Venkatesan, V., Saha, P., Vasudevan, G., Balakumar, G., & Balu, G. P. (2026) · IEEE Annual Computing and Communication Workshop and Conference (CCWC) · doi:10.1109/ccwc67433.2026.11393771
Bonifazi, G., Buratti, C., Marchetti, M., Parlapiano, F., Traini, D., Ursino, D., & Virgili, L. (2026) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3803797 · Cited by 1
“We evaluated GradQuiz’s performance using two MCQ datasets, i.e., OpenTriviaQA 1 [14, 29, 32, 35] and MMLU [23, 25, 27, 39].”
HaS: Accelerating RAG Through Homology-Aware Speculative Retrieval
Peng, P., Lin, W., Wu, W., Wang, X., & Liu, Y. (2026) · IEEE International Conference on Data Engineering (ICDE) · doi:10.1109/icde65706.2026.00143 · Cited by 1
Revisiting Disaggregated Large Language Model Serving for Performance and Energy Implications
Li, J., Zhu, Y., Chen, B., Lee, E. K., & Nahrstedt, K. (2026) · Workshop on Machine Learning and Systems (EuroMLSys) · doi:10.1145/3805621.3807662 · Cited by 4
“Through concrete case studies, we demonstrate that once the latency SLO is satisfied, energy–throughput efficiency can be further improved by independently tuning the operating frequencies of the prefill and decode GPUs.”
Dealing with the Elephant in the KV Cache: Video Frame Sampling for Multimodal LLM Inference
Papaioannou, K., & Doudali, T. D. (2026) · Workshop on Machine Learning and Systems (EuroMLSys) · doi:10.1145/3805621.3807613 · Cited by 1
RAG-Cache: Efficient Query-Specific Knowledge Caching for Retrieval Augmented Generation
Jin, H., & Feng, H. (2026) · Lecture notes in computer science · doi:10.1007/978-981-92-0378-9_46
VLMCache: Efficient On-Device Vision-Language Model Inference
Zhang, Y., Xu, D., Chen, Z., Wang, C., Zhang, Y., Xu, M., & Huang, G. (2026) · International Conference on Mobile Systems, Applications, and Services · doi:10.1145/3745756.3809243 · Cited by 3
RAPO: Retrieval-Augmented Phase Ordering
Yang, J., Lee, J.-H., Hong, Y., & Sung, H. (2026) · ACM SIGPLAN/SIGBED Conference on Languages, Compilers, and Tools for Embedded Systems (LCTES) · doi:10.1145/3814943.3816172
“Recent work on semantic caching [18, 23, 38, 66] and retrieval-augmented generation (RAG) for LLMs [13, 36, 47] shows that integrating a data-base of relevant prior data into the inference process and reusing results for semantically similar queries can substantially lower cost and latency.”
CONCORD: Asynchronous Sparse Aggregation for Device-Cloud Rag Under Document Isolation
Hu, X., Tang, Z., Yao, Z., Wang, T., & Weijia Jia, B. (2026) · IEEE International Conference on Web Services (ICWS) · doi:10.1109/icws72778.2026.00134
“RAGCache and Block-Attention further reduced the serving overhead of long retrieved context through cache reuse and more efficient prefilling [21], [22].”
Load balancing for large language model inference serving: A control and learning oriented survey
Ma, H.-B., Li, Z.-G., Wang, J.-N., Cao, C.-Y., Li, D.-L., Yang, J., & Ding, Z.-T. (2026) · Neurocomputing · doi:10.1016/j.neucom.2026.134973
HeteroRAGCache: Software-Hardware Co-Design for Efficient RAG Caching using Emerging Memories
Park, J., Suh, K., Ponzina, F., & Rosing, T. (2026) · ACM Great Lakes Symposium on VLSI (GLSVLSI) · doi:10.1145/3787109.3815268
Co-Optimizing Request Scheduling and KV Caching for Edge LLM Serving
Li, X., Jiao, W., He, J., & Zhang, S. (2026) · IEEE Internet of Things Journal · doi:10.1109/jiot.2026.3709703
Cache-Enhanced RAG and Graph-RAG for O-RAN
Bano, S., Nezami, Z., Hafeez, M., Zaidi, S. A., & Ahmed, Q. (2025) · IEEE Globecom Workshops (GC Wkshps) · doi:10.1109/gcwkshps68340.2025.11591230
“A promising solution to this problem lies in the use of cache memory for faster access to frequently retrieved data [11], [12] and reduced computational load.”
Omnia: Efficient RAG Serving through Speculative Scheduling
Fu, R., Li, S., Xu, Y., Wu, T., Ma, Z., & Shi, J. (2026) · ACM International Symposium on High-Performance Parallel and Distributed Computing (HPDC) · doi:10.1145/3806645.3807814 · Cited by 1
InKubeator: Pre-warming In-Memory KV Caches from Disk for Elastic LLM Serving
Muthukumar, A., Gough, E., & Nadig, D. (2026) · International Conference on Communications · doi:10.1109/icc59461.2026.11587970
“This issue is especially pronounced in long-context workloads, including Retrieval-Augmented Generation (RAG), where large documents must be processed before the initial token is produced [5].”
InfluentialReinforcement Learning for Adaptive Cache Policies in Retrieval-Augmented Generation Systems
Thatha, S., & Mudenagudi, U. (2025) · IEEE Conference on Engineering Informatics (ICEI) · doi:10.1109/icei68356.2025.11603697
“Jin et al. [13] introduced RAGCache, a framework that reduces redundant retrievals in RAG pipelines, though it employs heuristic similarity thresholds rather than adaptive learning.”
Non-Redundant and Hotness-Aware Caching for Low-Latency Retrieval-Augmented Generation
Xie, Z., Wang, H., Zhang, Y.-A., & Jin, P. (2026) · IEEE International Conference on Distributed Computing Systems Workshops (ICDCSW) · doi:10.1109/icdcsw72724.2026.00013
“Query skew in RAG refers to the highly uneven distribution of user queries, a characteristic prevalent in real-world search and conversational AI systems [3], [4], [6], [7].”
InfluentialMeridian: In-Memory Acceleration for RAG with Document Attention Decomposition
Liu, C., Huang, Y., Liu, H., Zhang, Y., Qihang, Q. I. U., Li, X., Zheng, L., Liao, X., Jin, H., & Xue, J. (2026) · Proceedings - International Symposium on Computer Architecture/Proceedings · doi:10.1109/isca66397.2026.00041
“Because attention scales quadratically with sequence length, prefilling these long contexts dominates computation and inflates Time-to-First-Token (TTFT), directly degrading the responsiveness of interactive applications [25].”
TS-Memory: Plug-and-Play Memory for Time Series Foundation Models
Lyu, S., Zhong, S., Chen, T., Ruan, W., Liu, Q., Lv, T., Wen, Q., Wong, R. C.-W., & Liang, Y. (2026) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3770855.3817614 · Cited by 2
Pick Up Where You Left Off: An Efficient Solution for Continuous Vector Similarity Search
Zheng, Z., Zeng, Y., Chi, Y., & Tong, Y. (2026) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3770855.3817657
Efficient VLM Inference System With LoRA Adapters at the Edge
Wang, W., Mi, L., Chen, J., Yu, T., Cao, T., Li, Y., Li, M., Dai, H., Chen, G., & Liu, Y. (2026) · IEEE Transactions on Mobile Computing · doi:10.1109/tmc.2026.3723365
EC-RAG: Towards Efficient Edge-Cloud Retrieval-Augmented Generation Systems
Wang, L., Wang, K., Jia, R., Lu, K., Wan, J., Huo, H., Zhai, Y., Liang, Z., & Wang, D. (2026) · IEEE International Conference on Data Engineering (ICDE) · doi:10.1109/icde65706.2026.00082
Pegasus: Accelerating Large Language Model Inference with Stateful Prefix Caching
Chen, F., Li, P., Yu, D., & Guo, S. (2026) · IEEE International Conference on Joint Cloud Computing (JCC) · doi:10.1109/jcc72984.2026.00017
“Crucially, unlike traditional caching systems where the benefit of a cache hit is uniform [4], [5], the computational savings in LLM inference are highly asymmetric due to the attention mechanism.”
InfluentialFlowCache: Workflow-Aware Intermediate-State Reuse for Multi-Tool LLM Serving
Wang, F., Li, M., Sheng, F., & Yang, K. (2026) · International Conference on Computer Science and Blockchain (CCSB) · doi:10.1109/ccsb71176.2026.11716474 · Cited by 1
“…and programmatic language-model workflows. vLLM [29], SGLang [30], and Prompt Cache [31] optimize token-and prefix-level state management; RAGCache [32] organizes retrieved knowledge state; and Mooncake, DistServe, ServerlessLLM, and Sarathi-Serve focus on disaggregation, cold starts, and…”
Edge-Cloud Collaborative RAG with Multi-State Semantic Caching
Liu, K., Xu, G., Fu, Y., Ding, M., Wu, Y., Li, J., & Yang, L. (2026) · Proceedings of the International Conference on Parallel Processing · doi:10.1145/3832810.3832878
Untariyati, N. A., Adi, K., Widodo, A. P., & Uliniansyah, M. T. (2026) · Research Square · doi:10.21203/rs.3.rs-9879191/v1
Dhakal, A., Pokharel, P., & Adhikari, S. (2026) · European Journal of Applied Science Engineering and Technology · doi:10.59324/ejaset.2026.4(5).11
Gudadhe, S., Rajankar, A., Jaggi, S., & Bhat, K. (2026) · International Conference on Computing Communication Control and automation · doi:10.1109/iccubea69821.2026.11701973
“…proposed to improve generation accuracy, such as R²AG proposed by Ye et al. [4] to improve generation accuracy, RAGCache proposed by Jin et al. [5] to improve FAISS pipelines, and domain-specific RAG, which is proposed by Sharma et al. [6] to reduce hallucination and improve factual…”
An Evidence-Grounded Multimodal AI Framework for Historical Digital Personae in Cultural Heritage
Lipianina-Honcharenko, K., & Yurkiv, K. (2026) · Big Data and Cognitive Computing · doi:10.3390/bdcc10100351
Anchor-NDP: An importance-based prefix KV management framework for CXL-enabled NDP
Kim, H., Kim, W., & Han, T. H. (2026) · Journal of Systems Architecture · doi:10.1016/j.sysarc.2026.104056
AdaCache: Efficient and Robust Noise-Resistant Semantic Caching for RAG
Chen, S., Zhang, L., & Lee, S.-W. (2026) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn69854.2026.11728681
“Subsequent methods such as vCache [11], PromptCache [21], RAGCache [9], and CRAG [22] improve reliability or retrieval efficiency, yet most treat caching as an isolated add-on and do not integrate retrieval logic [23] or internal-state signals for storage prioritization.”
InfluentialCacheSolidarity: Preventing Prefix Caching Side Channels in Multi-Tenant LLM Serving Systems
Pennas, P. G., Papaioannou, K., Guarnieri, M., & Doudali, T. D. (2026) · arXiv · doi:10.48550/arxiv.2603.10726
Metadata
- Authors
- Jin, C., Zhang, Z., Jiang, X., Liu, F., Liu, S., Liu, X., & Jin, X.
- Affiliations
- Peking University (CN)
- ByteDance (China) (CN)
- Venue
- ACM Transactions on Computer Systems
- Published
- 2025-11-07
- Volume
- 44
- Issue
- 1
- Pages
- 1-27
- ISSN
- 0734-2071 (ISSN-L)
- 1557-7333
- OpenAlex
- W4414371540
- Cited by
- 61
- References
- 26
- Citations by year
- 2026: 37 · 2025: 22
- FWCI
- 23.66
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- ACM Transactions on Computer Systems (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Key Research and Development Program of China (2022YFB4500700)
- Scientific Research Innovation Capability Support Project for Young Faculty (ZYGXQNJSKYCXNLZCXM-I1)
- Fundamental Research Funds for the Central Universities, Peking University, and the National Natural Science Foundation of China (62172008 and 62325201)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
- Keywords
- retrieval-augmented generation, latency optimization, large language models, tree of knowledge, GPU memory hierarchy, time-to-first-token, vLLM, FAISS
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.