Minimizing Response Latency in LLM-Based Agent Systems: A Comprehensive Survey
Open accessAbstract
The advent of Large Language Model (LLM)-based agent systems represents a significant paradigm shift in Artificial Intelligence, enabling unprecedented capabilities in autonomous reasoning, planning, and interaction. However, the transformative potential of these agents is fundamentally constrained by a critical operational challenge: high response latency. This latency, which impairs usability, restricts real-time applicability, and challenges economic viability, is a paramount barrier to their widespread adoption in complex, real-world scenarios. This survey provides a comprehensive and structured analysis of the multifaceted latency problem in LLM-based agent systems. We present a structured taxonomy that deconstructs end-to-end latency into its constituent sources, offering a holistic view of the entire agent stack. Our review spans four primary layers of optimization: (1) Core LLM Inference, covering techniques to accelerate the foundational model, including quantization, pruning, efficient attention mechanisms, and speculative decoding; (2) Agent-Level Frameworks, detailing strategies to refine the agent’s cognitive loop, such as accelerating planning cycles, optimizing tool use, and implementing latency-aware memory management; (3) System-Level and Infrastructure, encompassing optimizations for the underlying deployment environment, including advanced serving systems, hardware acceleration, dynamic resource allocation, and distributed architectures; and (4) Multi-Agent…
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Retrieval, routing, parsing, and tool invocation all introduce latency.
“Retrieval, routing, parsing, and tool invocation all introduce latency [Park et al., 2026, Xu et al., 2025a].”
From Externalization in LLM Agents: A Unified Review of Memory, Skills, Protocols and Harness Engineering (2026) · cites this paper as Park et al., 2026
Cites
A survey on large language model based autonomous agents
Wang, L., Ma, C., Feng, X., Zhang, Z., Yang, H., Zhang, J., Chen, Z., Tang, J., Chen, X., Lin, Y., et al. (2024) · Frontiers of Computer Science · doi:10.1007/s11704-024-40231-1 · Cited by 1,941 · see citing works
Efficient Memory Management for Large Language Model Serving with PagedAttention
Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez, J., Zhang, H., & Stoica, I. (2023) · Proceedings of the 29th Symposium on Operating Systems Principles · doi:10.1145/3600006.3613165 · Cited by 1,817 · cited 8× · see citing works
“Production-grade serving frameworks are essential for deploying optimized LLM inference at scale. vLLM [64] provides state-of-the-art inference serving with PagedAt-tention for efficient KV cache management (discussed in Section III-H), achieving 2-4 × throughput improvements over previous systems.”
InfluentialEfficient Memory Management for Large Language Model Serving with PagedAttention
Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez, J. E., Zhang, H., & Stoica, I. (2023) · arXiv · doi:10.48550/arxiv.2309.06180 · Cited by 1,802 · see citing works
ChatDev: Communicative Agents for Software Development
Qian, C., Liu, W., Liu, H., Chen, N., Dang, Y., Li, J., Yang, C., Chen, W., Su, Y., Cong, X., et al. (2024) · Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2024.acl-long.810 · Cited by 490 · see citing works
DeepSpeed- Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scale
Aminabadi, R. Y., Rajbhandari, S., Awan, A. A., Li, C., Li, D., Zheng, E., Ruwase, O., Smith, S., Zhang, M., Rasley, J., et al. (2022) · SC22: International Conference for High Performance Computing, Networking, Storage and Analysis · doi:10.1109/sc41404.2022.00051 · Cited by 297 · see citing works
He, J., Treude, C., & Lo, D. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3712003 · Cited by 246 · see citing works
Reasoning with Language Model is Planning with World Model
Hao, S., Gu, Y., Ma, H., Hong, J., Wang, Z., Wang, D., & Hu, Z. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.507 · Cited by 168 · see citing works
A Survey on the Memory Mechanism of Large Language Model-based Agents
Zhang, Z., Dai, Q., Bo, X., Ma, C., Li, R., Chen, X., Zhu, J., Dong, Z., & Wen, J.-R. (2025) · ACM Transactions on Information Systems · doi:10.1145/3748302 · Cited by 118 · see citing works
ChunkAttention: Efficient Self-Attention with Prefix-Aware KV Cache and Two-Phase Partition
Ye, L., Tao, Z., Huang, Y., & Li, Y. (2024) · arXiv · doi:10.48550/arxiv.2402.15220 · Cited by 34 · see citing works
Agashe, S., Fan, Y., Reyna, A., & Wang, X. E. (2025) · Findings of the Association for Computational Linguistics: NAACL 2025 · doi:10.18653/v1/2025.findings-naacl.448 · Cited by 32 · see citing works
Kong, Y., Ruan, J., Chen, Y., Zhang, B., Bao, T., Shiwei, S., Qing, D. G., Hu, X., Mao, H., Li, Z., et al. (2024) · Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: Industry Track · doi:10.18653/v1/2024.emnlp-industry.27 · Cited by 17 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Externalization in LLM Agents: A Unified Review of Memory, Skills, Protocols and Harness Engineering
Zhou, C., Chai, H., Chen, W., Guo, Z., Shan, R., Song, Y., Xu, T., Yang, Y., Yu, A., Zhang, W., et al. (2026) · arXiv · doi:10.48550/arxiv.2604.08224 · Cited by 1 · see citing works
- Authors
- Park, G., Lee, S., & Park, Y.
- Affiliations
- Kangwon National University (KR)
- Kyungpook National University (KR)
- Venue
- IEEE Access
- Published
- 2026
- Volume
- 14
- Pages
- 26140-26168
- ISSN
- 2169-3536 (ISSN-L)
- OpenAlex
- W7128709910
- Cited by
- 4
- References
- 11
- Citations by year
- 2026: 4
- FWCI
- 9.28
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Locations
- IEEE Access (publishedVersion, cc-by)
- Funders
- National Research Foundation of Korea (NRF) grant funded by Korean Government (RS-2023-00242528, RS-2025-00513098)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Artificial Intelligence Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- LLM agents, response latency, latency reduction, LLM inference acceleration, quantization, pruning, efficient attention mechanisms, speculative decoding, planning optimization, memory management, hardware acceleration, dynamic resource allocation, distributed architecture, multi-agent systems, hardware-software co-design
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
