TimelyLLM: Time-sensitive LLM Serving System for Physical-I/O Limited Agents

Abstract

Large Language Models (LLMs) are increasingly integrated into Physical-I/O limited agents, such as robots and voice assistants, which execute outputs sequentially. However, existing LLM serving systems typically employ a throughput-oriented batching mechanism, ignoring the large gap between LLM generation speed and the constrained physical I/O rates of agents, thus wasting execution slack and worsening resource contention. Besides, they treat all tokens equally and cannot anticipate the execution implications of different content, preventing scheduling aligned with agent-side behavior. To address it, we propose a new system named TimelyLLM that coordinates LLM generation with the physical behavior of agents. TimelyLLM introduces a novel segmented generation and scheduling mechanism, strategically leveraging the time gap between agent plan generation and execution to reduce contention and improve response latency under multi-agent workloads. We implement TimelyLLM on top of a widely-used LLM serving framework. We also build a dataset collection system to construct serving workloads from real-world robots, including drones, robot arms, and quadruped robots. Our evaluation demonstrates that TimelyLLM improves the time utility up to 1.52×, and reduces the overall waiting time by 84%.