InferCept: Efficient Intercept Support for Augmented Large Language Model Inference
Open accessAbstract
Large language models are increasingly integrated with external environments, tools, and agents like ChatGPT plugins to extend their capability beyond language-centric tasks. However, today's LLM inference systems are designed for standalone LLMs. They treat each external interaction as the end of LLM generation and form a new request when the interaction finishes, causing unnecessary recomputation of already computed contexts, which accounts for 37-40% of total model forwarding time. This paper presents InferCept, the first LLM inference framework targeting augmented LLMs and supporting the efficient interception of LLM generation. InferCept minimizes the GPU resource waste caused by LLM interceptions and dedicates saved memory for serving more requests. InferCept improves the overall serving throughput by 1.6x-2x and completes 2x more requests per second compared to the state-of-the-art LLM inference systems.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
InferCept makes its KV preserve decision based solely on reload cost, does not model per-turn queueing delay, and lacks a robust mechanism for variable tool call durations, making it impractical for real-world deployment.
“InferCept [2] makes its KV preserve decision based solely on the reload cost. But it does not model the per-turn queueing delay that accumulates over turns, nor have a robust mechanism to handle variable tool call durations. This makes it impractical for real-world deployment.”
From Continuum: Efficient and Robust Multi-Turn LLM Agent Scheduling with KV Cache Time-to-Live (2025) · cites this paper as [2]
Section 4.3 says InferCept chooses whether to preserve or discard KV context by comparing estimated memory waste, not solely reload cost. Section 4.4 explicitly proposes a dynamic duration estimate for highly variable interceptions and reports that it achieves 93% of oracle-based performance. The paper describes FCFS scheduling of waiting requests, though these passages do not establish whether it models per-turn queueing delay; they do not support the conclusion that InferCept is impractical for deployment. (LLM, conf=0.88)
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredInferCept's preserve operation is fixed and cannot adapt to tool use in real time; if the actual tool call time is much longer than predicted, blindly preserving the KV cache can cause significant inefficiency.
“Moreover, their preserve operation is fixed and could not adapt to tool use in real time. If the actual tool call time is much longer than predicted, blindly”
From Continuum: Efficient and Robust Multi-Turn LLM Agent Scheduling with KV Cache Time-to-Live (2025) · cites this paper as [2]
Section 4.4 (page unavailable) says InferCept updates its interception-duration estimate each iteration, contrary to the claim that its preserve decision is fixed and cannot adapt in real time. Section 3.2 does support the narrower point that preserving a paused request wastes GPU memory in proportion to the interception’s duration. (LLM, conf=0.88)
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredInferCept's preserve operation is invoked only when CPU offloading cost exceeds estimated GPU occupation cost during the tool call, and this decision ignores the queueing delay an evicted program experiences when it re-enters the waiting queue.
“InferCept [2]'s KV 'preserve' operation is invoked only when the CPU offloading cost exceeds the estimated GPU occupation cost during the tool call. Crucially, this decision only accounts for the reload cost of the immediate next turn-it entirely ignores the queueing delay that an evicted program will experience when it re-enters the waiting queue behind other active requests.”
From Continuum: Efficient and Robust Multi-Turn LLM Agent Scheduling with KV Cache Time-to-Live (2025) · cites this paper as [2]
Split into 2 atoms — 0 confirmed, 1 partially supported, 1 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredOutofOrderCost is the key term absent from prior retention policies such as InferCept, which only considers reload cost.
“This OutofOrderCost component is the key term absent from prior retention policies such as InferCept [2], which only considers the reload cost.”
From Continuum: Efficient and Robust Multi-Turn LLM Agent Scheduling with KV Cache Time-to-Live (2025) · cites this paper as [2]
Section 4.3 describes InferCept choosing among preserve, swap, and discard using estimated GPU-memory waste, then ordering intercepted requests by that waste. Section 3.2 also includes the effect of recomputation on other running requests. Thus, the paper contradicts the claim that InferCept considers only reload cost; it does not use the term “OutofOrderCost.” Page numbers were not available in our copy. (LLM, conf=0.88)
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredInferCept's policy overlooks the multi-turn nature of requests; when KV cache is evicted between turns, additional queueing time per turn accumulates for the program.
“However, their policy overlooks the multiturn nature of requests. When KV cache is evicted between turn, this will cause additional queueing time per turn for the program when they come back. In multi-turn scenarios, the queueing time can accumulate for each turn.”
From Continuum: Efficient and Robust Multi-Turn LLM Agent Scheduling with KV Cache Time-to-Live (2025) · cites this paper as [2]
The paper does not overlook multi-turn requests: §2.2 describes chatbot conversations as a sequence of chats whose history remains the context, and §4.3 defines a decision for each interception of a request. Section 4.3 also schedules resumed requests using their original arrival times. Our copy does not report the claimed accumulation of queueing time across turns. (LLM, conf=0.86)
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measured
API Serve and LLM-SQL study KV cache reuse for specific applications such as interleaving with external API calls and relational databases, but they do not have our radix tree or cache-aware scheduling.
“API Serve [1] and LLM-SQL [29] study KV cache reuse for specific applications such as interleaving with external API calls and relational databases, but they do not have our radix tree or cache-aware scheduling.”
From SGLang: Efficient Execution of Structured Language Model Programs (2023) · cites this paper as [1]
The Introduction and Additional information describe preserving or swapping the KV cache across external API calls, supporting the API-interleaving part of the claim. The paper does not discuss relational databases or the other cited system, LLM-SQL. Its Introduction and Section 3.1 also describe scheduling based on memory waste from intercepted requests, so the broad suggestion that it lacks cache-aware scheduling is not supported by our copy; whether it lacks the citing authors’ particular scheduler cannot be determined from this source. (LLM, conf=0.82)
Experimental — accuracy not yet measuredDuring external pauses, tool-augmented and coding agents require cache retention, offloading, eviction, and reload.
“During external pauses, tool-augmented and coding agents also require cache retention, offloading, eviction, and reload [70], [71], [72].”
From Unified AI Gateway: A Framework for Joint Model Routing and KV Cache Management (2026) · cites this paper as [70]
The Introduction describes external-tool pauses during which KV caches are temporarily unused but needed when decoding resumes. Section 4.3 describes preserving, swapping out, or discarding paused requests’ context, and the Swap-based approach describes swapping it back in. These are alternative strategies, not four operations every request requires; our copy does not specifically establish the claim for coding agents. (LLM, conf=0.84)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
High-Resolution Image Synthesis with Latent Diffusion Models
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., & Ommer, B. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01042 · Cited by 15,880 · cited 1× · see citing works
“To understand this use case, we use ChatGPT to create a dataset by generating a series of image-generation prompts, each triggering a call to the Stable Diffusion model (Rombach et al., 2021).”
MethodLLaMA: Open and Efficient Foundation Language Models
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. (2023) · arXiv · doi:10.48550/arxiv.2302.13971 · Cited by 4,602 · see citing works
Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q., & Zhou, D. (2022) · arXiv · doi:10.48550/arxiv.2201.11903 · Cited by 4,263 · cited 1× · see citing works
“For example, Chain-of-Thought (Wei et al., 2023; Zhang et al., 2023) breaks one request into a chain of “thought” steps, each being a new prompt to an LLM that continues the history of thoughts.”
BackgroundNo Language Left Behind: Scaling Human-Centered Machine TranslationPreprint
Costa-jussà, M., Cross, J., Çelebi, O., Elbayad, M., Heafield, K., Heffernan, K., Kalbassi, E., Lam, J., Licht, D., Maillard, J., et al. (2022) · arXiv.org · doi:10.48550/arxiv.2207.04672 · Cited by 2,176 · see citing works
A survey on large language model based autonomous agents
Wang, L., Ma, C., Feng, X., Zhang, Z., Yang, H., Zhang, J., Chen, Z., Tang, J., Chen, X., Lin, Y., et al. (2024) · Frontiers of Computer Science · doi:10.1007/s11704-024-40231-1 · Cited by 1,941 · see citing works
Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena
Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E. P., et al. (2023) · arXiv · doi:10.48550/arxiv.2306.05685 · Cited by 1,888 · cited 5× · see citing works
“We use the ShareGPT dataset (Zheng et al., 2023a), which contains crowd-sourced real-user ChatGPT conversations to evaluate chatbots.”
BackgroundMethodInfluentialSelf-citationEfficient Memory Management for Large Language Model Serving with PagedAttention
Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez, J., Zhang, H., & Stoica, I. (2023) · Proceedings of the 29th Symposium on Operating Systems Principles · doi:10.1145/3600006.3613165 · Cited by 1,817 · cited 7× · see citing works
“This GPU memory could accommodate other requests, which increases throughput (Kwon et al., 2023).”
BackgroundMethodInfluentialSelf-citationHotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering
Yang, Z., Qi, P., Zhang, S., Bengio, Y., Cohen, W., Salakhutdinov, R., & Manning, C. D. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1259 · Cited by 1,804 · cited 2× · see citing works
“While having longer context lengths for questions, these QA tools provide relatively quick yet variable execution times due to the non-deterministic network communication latency with Wikipedia.”
MethodMegatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism
Shoeybi, M., Patwary, M., Puri, R., LeGresley, P., Casper, J., & Catanzaro, B. (2019) · arXiv · doi:10.48550/arxiv.1909.08053 · Cited by 893 · cited 1× · see citing works
“We evaluate the 13B model with tensor-parallelism (Shoeybi et al., 2019) on two GPUs.”
MethodReAct: Synergizing Reasoning and Acting in Language Models
Yao, S., Zhao, J., Yu, D., Du, N., Shafran, I., Narasimhan, K., & Cao, Y. (2022) · arXiv · doi:10.48550/arxiv.2210.03629 · Cited by 608 · cited 1× · see citing works
“Similarly, Re-Act (Yao et al., 2023) uses a chain of reasoning and action steps to accomplish complex tasks.”
BackgroundHow many words do we read per minute? A review and meta-analysis of reading ratePreprint
Brysbaert, M. (2019) · Journal of Memory and Language · doi:10.31234/osf.io/xynwg · Cited by 512 · cited 1× · see citing works
“We estimate human response time ( i.e. , the interception time) as the summation of human scanning time (Brysbaert, 2019) of the previous chat response and typing time (Ma et al., 2015) of the next prompt.”
BackgroundHuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Face
Shen, Y., Song, K., Tan, X., Li, D., Lu, W., & Zhuang, Y. (2023) · arXiv · doi:10.48550/arxiv.2303.17580 · Cited by 477 · cited 1× · see citing works
“Instead of fine-tuning, another approach is instructing LLMs to select tools or models for a user task (Shen et al., 2023; Lu et al., 2023).”
MethodLanguage Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agents
Huang, W., Abbeel, P., Pathak, D., & Mordatch, I. (2022) · arXiv · doi:10.48550/arxiv.2201.07207 · Cited by 304 · cited 1× · see citing works
“An LLM can be augmented to interact with virtual environments and accomplish complex tasks by performing gradual steps (Gu et al., 2022; Huang et al., 2022; Hu et al., 2023).”
BackgroundSplitwise: Efficient Generative LLM Inference Using Phase Splitting
Patel, P., Choukse, E., Zhang, C., Shah, A., Goiri, Í., Maleki, S., & Bianchini, R. (2024) · 2024 ACM/IEEE 51st Annual International Symposium on Computer Architecture (ISCA) · doi:10.1109/isca59077.2024.00019 · Cited by 298 · cited 1× · see citing works
“These metrics are important in meeting service-level objectives and system response time (Zhong et al., 2024; Hu et al., 2024; Patel et al., 2024).”
BackgroundDeepSpeed- Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scale
Aminabadi, R. Y., Rajbhandari, S., Awan, A. A., Li, C., Li, D., Zheng, E., Ruwase, O., Smith, S., Zhang, M., Rasley, J., et al. (2022) · SC22: International Conference for High Performance Computing, Networking, Storage and Analysis · doi:10.1109/sc41404.2022.00051 · Cited by 297 · see citing works
Automatic Chain of Thought Prompting in Large Language Models
Zhang, Z., Zhang, A., Li, M., & Smola, A. (2022) · arXiv · doi:10.48550/arxiv.2210.03493 · Cited by 274 · cited 1× · see citing works
“For example, Chain-of-Thought (Wei et al., 2023; Zhang et al., 2023) breaks one request into a chain of “thought” steps, each being a new prompt to an LLM that continues the history of thoughts.”
BackgroundViperGPT: Visual Inference via Python Execution for Reasoning
Surís, D., Menon, S., & Vondrick, C. (2023) · 2023 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01092 · Cited by 273 · cited 1× · see citing works
“Other efforts enable LLMs to generate compositions of augmented LLMs (Suris et al., 2023; Qian et al., 2023).”
BackgroundLook Before You Leap: Unveiling the Power of GPT-4V in Robotic Vision-Language PlanningPreprint
Hu, Y.-D., Lin, F., Zhang, T., Yi, L., & Gao, Y. (2023) · arXiv.org · doi:10.48550/arxiv.2311.17842 · Cited by 260 · cited 2× · see citing works
“An LLM can be augmented to interact with virtual environments and accomplish complex tasks by performing gradual steps (Gu et al., 2022; Huang et al., 2022; Hu et al., 2023).”
BackgroundFast Distributed Inference Serving for Large Language ModelsPreprint
Wu, B., Zhong, Y.-M., Zhang, Z.-L., Liu, S.-Y., Liu, F.-Y., Sun, Y.-H., Huang, G., Liu, X.-Z., & Jin, X. (2023) · arXiv.org · doi:10.48550/arxiv.2305.05920 · Cited by 248 · see citing works
Gorilla: Large Language Model Connected with Massive APIs
Patil, S., Zhang, T., Wang, X., & Gonzalez, J. (2024) · Advances in Neural Information Processing Systems 37 · doi:10.52202/079017-4020 · Cited by 226 · cited 1× · see citing works
“Frameworks such as LangChain (Chase, 2022), DSpy (Khattab et al., 2024), Gorilla (Patil et al., 2023), SGLang (Zheng et al., 2023b), and AgentGraph (Chen et al., 2019) provide programming models for users to write their own flows of augmented LLMs.”
BackgroundVision-and-Language Navigation: A Survey of Tasks, Methods, and Future Directions
Gu, J., Stefani, E., Wu, Q., Thomason, J., & Wang, X. (2022) · Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2022.acl-long.524 · Cited by 216 · cited 1× · see citing works
“An LLM can be augmented to interact with virtual environments and accomplish complex tasks by performing gradual steps (Gu et al., 2022; Huang et al., 2022; Hu et al., 2023).”
Background(2024) · doi:10.1145/3704435 · Cited by 184 · cited 1× · see citing works
“…to handle more diverse and open-ended tasks, a recent trend is to augment LLMs with external tools (Mialon et al., 2023; Wang et al., 2023; Qin et al., 2023) such as arithmetic calculation (Hao et al., 2023) and ChatGPT plugins (OpenAI, 2023), non-LLM models like image generation (Betker…”
BackgroundSGLang: Efficient Execution of Structured Language Model Programs
Zheng, L., Yin, L., Xie, Z., Sun, C., Huang, J., Yu, C. H., Cao, S., Kozyrakis, C., Stoica, I., Gonzalez, J. E., et al. (2023) · arXiv · doi:10.48550/arxiv.2312.07104 · Cited by 145 · cited 1× · see citing works
“Frameworks such as LangChain (Chase, 2022), DSpy (Khattab et al., 2024), Gorilla (Patil et al., 2023), SGLang (Zheng et al., 2023b), and AgentGraph (Chen et al., 2019) provide programming models for users to write their own flows of augmented LLMs.”
BackgroundALFWorld: Aligning Text and Embodied Environments for Interactive Learning
Shridhar, M., Yuan, X., Côté, M.-A., Bisk, Y., Trischler, A., & Hausknecht, M. (2020) · arXiv · doi:10.48550/arxiv.2010.03768 · Cited by 80 · cited 2× · see citing works
“…et al., 2023; Qin et al., 2023) such as arithmetic calculation (Hao et al., 2023) and ChatGPT plugins (OpenAI, 2023), non-LLM models like image generation (Betker et al., 2024), and interaction with humans and other environments in real-time, such as virtual environments (Shridhar et al., 2021).”
BackgroundMethodHaptic keyclick feedback improves typing speed and reduces typing errors on a flat keyboard
Ma, Z., Edge, D., Findlater, L., & Tan, H. Z. (2015) · World Haptics Conference · doi:10.1109/whc.2015.7177717 · Cited by 51 · cited 1× · see citing works
“We estimate human response time ( i.e. , the interception time) as the summation of human scanning time (Brysbaert, 2019) of the previous chat response and typing time (Ma et al., 2015) of the next prompt.”
BackgroundToolkenGPT: Augmenting Frozen Language Models with Massive Tools via Tool Embeddings
Hao, S., Liu, T., Wang, Z., & Hu, Z. (2023) · arXiv · doi:10.48550/arxiv.2305.11554 · Cited by 35 · cited 5× · see citing works
“…a recent trend is to augment LLMs with external tools (Mialon et al., 2023; Wang et al., 2023; Qin et al., 2023) such as arithmetic calculation (Hao et al., 2023) and ChatGPT plugins (OpenAI, 2023), non-LLM models like image generation (Betker et al., 2024), and interaction with humans and…”
BackgroundMethodInfluentialTALM: Tool Augmented Language Models
Parisi, A., Zhao, Y., & Fiedel, N. (2022) · arXiv · doi:10.48550/arxiv.2205.12255 · Cited by 35 · cited 1× · see citing works
“Previous research (Parisi et al., 2022; Hao et al., 2023; Schick et al., 2023) has shown the effectiveness of fine-tuning an LLM to generate appropriate tool calls automatically during the decoding phase.”
MethodCREATOR: Tool Creation for Disentangling Abstract and Concrete Reasoning of Large Language Models
Qian, C., Han, C., Fung, Y., Qin, Y., Liu, Z., & Ji, H. (2023) · Findings of the Association for Computational Linguistics: EMNLP 2023 · doi:10.18653/v1/2023.findings-emnlp.462 · Cited by 24 · cited 2× · see citing works
“…to handle more diverse and open-ended tasks, a recent trend is to augment LLMs with external tools (Mialon et al., 2023; Wang et al., 2023; Qin et al., 2023) such as arithmetic calculation (Hao et al., 2023) and ChatGPT plugins (OpenAI, 2023), non-LLM models like image generation (Betker…”
BackgroundSARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefills
Agrawal, A., Panwar, A., Mohan, J., Kwatra, N., Gulavani, B. S., & Ramjee, R. (2023) · arXiv · doi:10.48550/arxiv.2308.16369 · Cited by 20 · cited 2× · see citing works
“Sarathi (Agrawal et al., 2023) is a system that proposes the technique of chunked prefill, which splits prompt tokens into chunks, each merged with other decoding requests to form one iteration’s batch.”
MethodInference without Interference: Disaggregate LLM Inference for Mixed Downstream Workloads
Hu, C., Huang, H., Xu, L., Chen, X., Xu, J., Chen, S., Feng, H., Wang, C., Wang, S., Bao, Y., et al. (2024) · arXiv · doi:10.48550/arxiv.2401.11181 · Cited by 8 · cited 1× · see citing works
“These metrics are important in meeting service-level objectives and system response time (Zhong et al., 2024; Hu et al., 2024; Patel et al., 2024).”
BackgroundPreble: Efficient Distributed Prompt Scheduling for LLM Serving
Srivatsa, V., He, Z., Abhyankar, R., Li, D., & Zhang, Y. (2024) · arXiv · doi:10.48550/arxiv.2407.00023 · Cited by 1 · see citing works
Self-citationOpenAI, Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.08774 · cited 1×
“Large language models (LLMs) (Achiam et al., 2023; Tou-vron et al., 2023) have shown immense potential in various applications, such as natural language understanding and content generation.”
Background
Flagged first, then by citations
18 of the 50 references couldn’t be matched to a known paper.
Research works printed but not linked: 10
- Ainslie, J., Lee-Thorp, J., de Jong, M., Zemlyanskiy, Y., Lebron, F., and Sanghai, S. GQA: Training generalized multi-query transformer models from multi-head check- points. In Proceedings of the 2023 Conference on Empir- ical Methods in Natural Language Processing (EMNLP 2023), Singapore, December 2023.
- Baeza-Yates, R., Ribeiro-Neto, B., et al. Modern Infor- mation Retrieval, volume 463. ACM Press, New York, 1999.
- Chen, L., Chen, Z., Tan, B., Long, S., Gasic, M., and Yu, K. Agentgraph: Towards universal dialogue manage- ment with structured deep reinforcement learning. arXiv preprint arXiv:1905.11259, May 2019. · arXiv
- Khattab, O., Singhvi, A., Maheshwari, P., Zhang, Z., San- thanam, K., A, S. V., Haq, S., Sharma, A., Joshi, T. T., Moazam, H., Miller, H., Zaharia, M., and Potts, C. DSPy: Compiling declarative language model calls into state-of-the-art pipelines. In The Twelfth International Conference on Learning Representations (ICLR '24), Vienna, Austria, 2024. URL https://openreview.net/ forum?id=sY5N0zY5Od.
- Li, Z., Zheng, L., Zhong, Y., Liu, V., Sheng, Y., Jin, X., Huang, Y., Chen, Z., Zhang, H., Gonzalez, J. E., and Stoica, I. AlpaServe: Statistical multiplexing with model parallelism for deep learning serving. In 17th USENIX Symposium on Operating Systems Design and Imple- mentation (OSDI 23), Boston, MA, July 2023.
- Lu, P., Peng, B., Cheng, H., Galley, M., Chang, K.-W., Wu, Y. N., Zhu, S.-C., and Gao, J. Chameleon: Plug-and-play compositional reasoning with large language models. In Proceedings of the 37th International Conference on Neural Information Processing Systems (NeurIPS '23), New Orleans, Louisiana, December 2023.
- Mialon, G., Dessi, R., Lomeli, M., Nalmpantis, C., Pa- sunuru, R., Raileanu, R., Roziere, B., Schick, T., Dwivedi-Yu, J., Celikyilmaz, A., Grave, E., LeCun, Y., and Scialom, T. Augmented language models: a survey. Transactions on Machine Learning Research (TMLR), 2023. ISSN 2835-8856. URL https://openreview.net/ forum?id=jh7wH2AzKK. Survey Certification.
- Qin, Y., Hu, S., Lin, Y., Chen, W., Ding, N., Cui, G., Zeng, Z., Huang, Y., Xiao, C., Han, C., Fung, Y. R., Su, Y., Wang, H., Qian, C., Tian, R., Zhu, K., Liang, S., Shen, X., Xu, B., Zhang, Z., Ye, Y., Li, B., Tang, Z., Yi, J., Zhu, Y., Dai, Z., Yan, L., Cong, X., Lu, Y., Zhao, W., Huang, Y., Yan, J., Han, X., Sun, X., Li, D., Phang, J., Yang, C., Wu, T., Ji, H., Liu, Z., and Sun, M. Tool learning with foundation models. arXiv preprint arXiv:2304.08354, June 2023. · arXiv
- Schick, T., Dwivedi-Yu, J., Dessì, R., Raileanu, R., Lomeli, M., Hambro, E., Zettlemoyer, L., Cancedda, N., and Scialom, T. Toolformer: Language models can teach themselves to use tools. 37th Conference on Neural In- formation Processing Systems, 2023.
- Yu, G.-I., Jeong, J. S., Kim, G.-W., Kim, S., and Chun, B.- G. Orca: A Distributed Serving System for Transformer- Based Generative Models. In 16th USENIX Sympo- sium on Operating Systems Design and Implementation (OSDI '22), Carlsbad, CA, July 2022.
Cited by
Sorted by citations
SGLang: Efficient Execution of Structured Language Model Programs
Zheng, L., Yin, L., Xie, Z., Sun, C., Huang, J., Yu, C. H., Cao, S., Kozyrakis, C., Stoica, I., Gonzalez, J. E., et al. (2023) · arXiv · doi:10.48550/arxiv.2312.07104 · Cited by 145 · see citing works
“…a simple table-based cache instead of a tree-structured cache, "FCFS Schedule" means using a first-come-first-serve policy instead of our cache-aware scheduling, "Random Schedule" means using a random order to schedule requests, "No Frontend Parallelism" means disabling parallelism in the…”
HotPrefix: Hotness-Aware KV Cache Scheduling for Efficient Prefix Sharing in LLM Inference Systems
Li, Y., Gu, R., Huan, C., Wang, Z., Yao, R., Tian, C., & Chen, G. (2025) · Proceedings of the ACM on Management of Data · doi:10.1145/3749168 · Cited by 9 · see citing works
Unified AI Gateway: A Framework for Joint Model Routing and KV Cache Management
Lu, J., Zhang, X., & Shao, Y. (2026) · arXiv · doi:10.48550/arxiv.2609.06940
“During external pauses, tool-augmented and coding agents also require cache retention, offloading, eviction, and reload [70], [71], [72].”
Continuum: Efficient and Robust Multi-Turn LLM Agent Scheduling with KV Cache Time-to-Live
Li, H., He, R., Mang, Q., Zhang, Q., Mao, H., Chen, X., Zhou, H., Zhang, H., Cheung, A., Gonzalez, J., et al. (2025) · arXiv · doi:10.48550/arxiv.2511.02230
“This is because the increased number of turns leads to more tool calls and longer overall execution times, exacerbating the scheduling challenges faced by traditional methods.”
InfluentialSelf-citationSmoothAgent: Efficient Long-Horizon LLM-Based Agent Serving with Lookahead Context Engineering
Pan, Z., Wang, Q., Hu, Z., Chen, C., Guan, Y., Zhou, Y., Swanson, S., & Ding, Y. (2026) · arXiv · doi:10.48550/arxiv.2607.00151
- Authors
- Abhyankar, R., He, Z., Srivatsa, V., Zhang, H., & Zhang, Y.
- Affiliations
- University of California San Diego (US)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2024
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2402.01869
- OpenAlex
- W4391591762
- Cited by
- 0
- References
- 50
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Hardware and Architecture › Parallel Computing and Optimization Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- large language model inference, GPU utilization, LLM agents, memory efficiency
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
