Towards End-to-End Optimization of LLM-based Applications with Ayo
Open accessPublished version of a preprint
Abstract
Large language model (LLM)-based applications consist of both LLM and non-LLM components, each contributing to the end-to-end latency. Despite great efforts to optimize LLM inference, end-to-end workflow optimization has been overlooked. Existing frameworks employ coarse-grained orchestration with task modules, which confines optimizations to within each module and yields suboptimal scheduling decisions.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Teola, Alto, and Parrot assume static or deterministically defined DAGs and cannot work with dynamic agent workloads like ReAct-styled ones whose dependency graphs evolve at runtime.
“One shared limitation of Teola, Parrot, and Alto is that they all assume static or deterministically defined DAGs and could not work with dynamic agent workloads like ReAct-styled ones whose dependency graphs evolve at runtime.”
From Continuum: Efficient and Robust Multi-Turn LLM Agent Scheduling with KV Cache Time-to-Live (2025) · cites this paper as [49]
Ayo, Alto, and Parrot assume static workflows and do not apply to dynamic agents.
“Ayo [49], Alto [46], and Parrot [34] assume static workflows and do not apply to dynamic agents.”
From Continuum: Efficient and Robust Multi-Turn LLM Agent Scheduling with KV Cache Time-to-Live (2025) · cites this paper as [49]
Cites
Efficient Memory Management for Large Language Model Serving with PagedAttention
Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez, J., Zhang, H., & Stoica, I. (2023) · Proceedings of the 29th Symposium on Operating Systems Principles · doi:10.1145/3600006.3613165 · Cited by 1,817 · see citing works
Efficient Memory Management for Large Language Model Serving with PagedAttention
Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez, J. E., Zhang, H., & Stoica, I. (2023) · arXiv · doi:10.48550/arxiv.2309.06180 · Cited by 1,802 · see citing works
FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness
Dao, T., Fu, D. Y., Ermon, S., Rudra, A., & Ré, C. (2022) · arXiv · doi:10.48550/arxiv.2205.14135 · Cited by 1,214 · see citing works
SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models
Xiao, G., Lin, J., Seznec, M., Wu, H., Demouth, J., & Han, S. (2022) · arXiv · doi:10.48550/arxiv.2211.10438 · Cited by 247 · see citing works
Miao, X., Oliaro, G., Zhang, Z., Cheng, X., Wang, Z., Zhang, Z., Wong, R. Y. Y., Zhu, A., Yang, L., Shi, X., et al. (2024) · Proceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 3 · doi:10.1145/3620666.3651335 · Cited by 114 · see citing works
Bang, F. (2023) · Proceedings of the 3rd Workshop for Natural Language Processing Open Source Software (NLP-OSS 2023) · doi:10.18653/v1/2023.nlposs-1.24 · Cited by 72 · cited 1× · see citing works
“…on various optimization directions, including kernel acceleration [30, 36, 73], request scheduling [24, 25, 65, 76], model parallelism [50, 60, 80], semantic cache [26, 81], KV cache management [48, 51, 71], KV cache reusing [33, 45, 48, 55, 79] and advanced decoding algorithms [56, 59, 62].”
An LLM Compiler for Parallel Function Calling
Kim, S., Moon, S., Tabrizi, R., Lee, N., Mahoney, M. W., Keutzer, K., & Gholami, A. (2023) · arXiv · doi:10.48550/arxiv.2312.04511 · Cited by 12 · see citing works
Optimizing LLM Queries in Relational Data Analytics Workloads
Liu, S., Biswal, A., Kamsetty, A., Cheng, A., Schroeder, L. G., Patel, L., Cao, S., Mo, X., Stoica, I., Gonzalez, J. E., et al. (2024) · arXiv · doi:10.48550/arxiv.2403.05821 · Cited by 8 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
SmoothAgent: Efficient Long-Horizon LLM-Based Agent Serving with Lookahead Context Engineering
Pan, Z., Wang, Q., Hu, Z., Chen, C., Guan, Y., Zhou, Y., Swanson, S., & Ding, Y. (2026) · arXiv · doi:10.48550/arxiv.2607.00151
“Parrot [37] and Ayo [61] model static agent workflows as directed acyclic graphs (DAGs) and apply a range of optimizations over the execution graph.”
Continuum: Efficient and Robust Multi-Turn LLM Agent Scheduling with KV Cache Time-to-Live
Li, H., He, R., Mang, Q., Zhang, Q., Mao, H., Chen, X., Zhou, H., Zhang, H., Cheung, A., Gonzalez, J., et al. (2025) · arXiv · doi:10.48550/arxiv.2511.02230
“Moreover, since tool calls can be inherently variable, we need to set a maximum KV cache retention time to prevent infinitely long waiting.”
- Authors
- Tan, X., Jiang, Y., Yang, Y., & Xu, H.
- Affiliations
- Chinese University of Hong Kong (HK)
- Unaffiliated, Beijing, China
- Venue
- Proceedings of the 30th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2
- Publisher
- ACM
- Published
- 2025-03-30
- Pages
- 1302-1316
- OpenAlex
- W4408901326
- Cited by
- 13
- References
- 8
- Citations by year
- 2026: 13
- FWCI
- 11.11
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) (publishedVersion, cc-by)
- Funders
- Research Grants Council of Hong Kong (C7004-22G)
- CUHK (4937007,4937008,5501329,5501517)
- Topics
- Physical Sciences › Computer Science › Hardware and Architecture › Parallel Computing and Optimization Techniques
- Physical Sciences › Computer Science › Computer Networks and Communications › Software System Performance and Reliability
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- end-to-end optimization, large language models, end-to-end delay, workflow scheduling, task orchestration, inference optimization, Ayo
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
