An LLM Compiler for Parallel Function Calling
Open accessAbstract
The reasoning capabilities of the recent LLMs enable them to execute external function calls to overcome their inherent limitations, such as knowledge cutoffs, poor arithmetic skills, or lack of access to private data. This development has allowed LLMs to select and coordinate multiple functions based on the context to tackle more complex problems. However, current methods for function calling often require sequential reasoning and acting for each function which can result in high latency, cost, and sometimes inaccurate behavior. To address this, we introduce LLMCompiler, which executes functions in parallel to efficiently orchestrate multiple function calls. Drawing inspiration from the principles of classical compilers, LLMCompiler enables parallel function calling with three components: (i) a Function Calling Planner, formulating execution plans for function calling; (ii) a Task Fetching Unit, dispatching function calling tasks; and (iii) an Executor, executing these tasks in parallel. LLMCompiler automatically generates an optimized orchestration for the function calls and can be used with both open-source and closed-source models. We have benchmarked LLMCompiler on a range of tasks with different patterns of function calling. We observe consistent latency speedup of up to 3.7x, cost savings of up to 6.7x, and accuracy improvement of up to ~9% compared to ReAct. Our code is available at https://github.com/SqueezeAILab/LLMCompiler.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
LM programs typically contain multiple LLM calls interspersed with control flow.
“All of these new use cases require multiple, often dependent, LLM generation calls, showing a trend of using multi-call structures to complete complex tasks [57, 21].”
From SGLang: Efficient Execution of Structured Language Model Programs (2023) · cites this paper as [21]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q., & Zhou, D. (2022) · arXiv · doi:10.48550/arxiv.2201.11903 · Cited by 4,263 · cited 1× · see citing works
“…reasoning capability of Large Language Models (LLMs) have expanded the applicability of LLMs beyond content generation to solving complex problems [51, 23, 57, 5, 50, 63, 8, 55, 12]; and recent works have shown how this reasoning capability could be helpful in improving accuracy for solving…”
BackgroundThe Power of Scale for Parameter-Efficient Prompt Tuning
Lester, B., Al-Rfou, R., & Constant, N. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.243 · Cited by 3,096 · cited 1× · see citing works
“However, there is a growing demand for using open-source models driven by the increasing number of open-source LLMs as well as parameter-efficient training techniques [27, 18, 17] for finetuning and customization.”
MethodDerman, E. (2009) · The SAGE Encyclopedia of Research Design · doi:10.2469/faj.v65.n1.5 · Cited by 2,590 · cited 2×
“This method involves a two-step process of an initial skeleton generation phase, followed by parallel execution of skeleton items.”
LoRA: Low-Rank Adaptation of Large Language Models
Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., & Chen, W. (2021) · arXiv · doi:10.48550/arxiv.2106.09685 · Cited by 2,523 · cited 1× · see citing works
“However, there is a growing demand for using open-source models driven by the increasing number of open-source LLMs as well as parameter-efficient training techniques [27, 18, 17] for finetuning and customization.”
MethodParameter-efficient transfer learning for NLP
Houlsby, N., Giurgiu, A., Jastrzebski, S., Morrone, B., de Laroussilhe, Q., Gesmundo, A., Attariyan, M., & Gelly, S. (2019) · arXiv · doi:10.48550/arxiv.1902.00751 · Cited by 2,270 · cited 1× · see citing works
“However, there is a growing demand for using open-source models driven by the increasing number of open-source LLMs as well as parameter-efficient training techniques [27, 18, 17] for finetuning and customization.”
MethodShirky, C. (1999) · doi:10.1145/323409.328679 · Cited by 2,043 · cited 2×
“We look forward to future work building upon our framework that will improve both the capabilities and efficiencies of LLMs in executing complex, large-scale tasks, thus transforming the future development of LLM-based applications.”
Efficient Memory Management for Large Language Model Serving with PagedAttention
Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez, J., Zhang, H., & Stoica, I. (2023) · Proceedings of the 29th Symposium on Operating Systems Principles · doi:10.1145/3600006.3613165 · Cited by 1,817 · cited 1× · see citing works
“Various studies have focused on optimizing model design [21, 11, 30, 9, 24, 10, 22, 6, 28] and systems [25, 59, 1, 2] for efficient LLM inference.”
BackgroundHotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering
Yang, Z., Qi, P., Zhang, S., Bengio, Y., Cohen, W., Salakhutdinov, R., & Manning, C. D. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1259 · Cited by 1,804 · cited 3× · see citing works
“Furthermore, while dynamic reasoning about the observations has benefits in certain cases, Figure 1: An illustration of the runtime dynamics of LLMCompiler , in comparison with ReAct [58], given a sample question from the HotpotQA benchmark [54].”
MethodInfluentialEvaluating Large Language Models Trained on Code
Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. D. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. (2021) · arXiv · doi:10.48550/arxiv.2107.03374 · Cited by 1,436 · cited 1× · see citing works
“This limits its applicability in complex scenarios such as coding [7, 33, 14, 3] or math [16, 15] problems, as also stated in the paper [34].”
BackgroundFralick, M., Sacks, C. A., Muller, D., Vining, T., Ling, E., Drazen, J., & Hardin, C. (2023) · Communications of the ACM · doi:10.1007/978-981-96-6259-3 · Cited by 1,397 · cited 1× · see citing works
“Gorilla [39] introduced a finetuned LLM designed for function calling, and ToolLLM [41] and RestGPT [46] have extended LLMs to support real-world APIs.”
BackgroundFEVER: a Large-scale Dataset for Fact Extraction and VERification
Thorne, J., Vlachos, A., Christodoulopoulos, C., & Mittal, A. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1074 · Cited by 1,343 · cited 1× · see citing works
“A notable work is ReWOO [53] which employs a planner to separate the reasoning process from the execution and observation phases to decrease token usage and cost as compared to ReAct.”
MethodTree of Thoughts: Deliberate Problem Solving with Large Language Models
Yao, S., Yu, D., Zhao, J., Shafran, I., Griffiths, T. L., Cao, Y., & Narasimhan, K. (2023) · arXiv · doi:10.48550/arxiv.2305.10601 · Cited by 1,158 · cited 5× · see citing works
“Here, we consider one such dynamic approach in the context of the Game of 24 with the Tree-of-Thoughts (ToT) strategy proposed in [57].”
BackgroundMethodResultInfluentialToolformer: Language Models Can Teach Themselves to Use Tools
Schick, T., Dwivedi-Yu, J., Dessì, R., Raileanu, R., Lomeli, M., Zettlemoyer, L., Cancedda, N., & Scialom, T. (2023) · arXiv · doi:10.48550/arxiv.2302.04761 · Cited by 1,150 · cited 1× · see citing works
“A notable work is Toolformer [43], which produces a custom LLM output to let the LLM decide what the inputs for calling the functions should be and where to insert the result.”
BackgroundPlan-and-Solve Prompting: Improving Zero-Shot Chain-of-Thought Reasoning by Large Language ModelsPreprint
Wang, L., Xu, W., Lan, Y.-H., Hu, Z.-Q., Lan, Y.-S., Lee, R. K.-W., & Lim, E.-P. (2023) · Annual Meeting of the Association for Computational Linguistics · doi:10.48550/arxiv.2305.04091 · Cited by 937 · cited 1× · see citing works
“Plan-and-Solve Prompting [49] segments multi-step reasoning tasks into subtasks to minimize errors and improve task accuracy without manual prompting.”
BackgroundDecomposed Prompting: A Modular Approach for Solving Complex TasksPreprint
Khot, T., Trivedi, H., Finlayson, M., Fu, Y., Richardson, K., Clark, P., & Sabharwal, A. (2022) · International Conference on Learning Representations · doi:10.48550/arxiv.2210.02406 · Cited by 852 · cited 2× · see citing works
“Specifically, Decomposed Prompting [20] tackles complex tasks by decomposing them into simpler sub-tasks, each optimized through LLMs with dedicated prompts.”
BackgroundSelf-Consistency Improves Chain of Thought Reasoning in Language Models
Wang, X., Wei, J., Schuurmans, D., Le, Q., Chi, E., Narang, S., Chowdhery, A., & Zhou, D. (2022) · arXiv · doi:10.48550/arxiv.2203.11171 · Cited by 829 · cited 1× · see citing works
“…of Large Language Models (LLMs) have expanded the applicability of LLMs beyond content generation to solving complex problems [51, 23, 57, 5, 50, 63, 8, 55, 12]; and recent works have shown how this reasoning capability could be helpful in improving accuracy for solving complex and logical…”
BackgroundReAct: Synergizing Reasoning and Acting in Language Models
Yao, S., Zhao, J., Yu, D., Du, N., Shafran, I., Narasimhan, K., & Cao, Y. (2022) · arXiv · doi:10.48550/arxiv.2210.03629 · Cited by 608 · cited 10× · see citing works
“4.1, we use ReAct [58] as the main baseline.”
BackgroundMethodInfluentialBeyond the Imitation Game: Quantifying and extrapolating the capabilities of language models
Srivastava, A., Rastogi, A., Rao, A., Shoeb, A. A. M., Abid, A., Fisch, A., Brown, A. R., Santoro, A., Gupta, A., Garriga-Alonso, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2206.04615 · Cited by 569 · cited 2× · see citing works
“• Movie Recommendation: A dataset with 500 examples that asks to identify the most similar movie out of four options to another set of four movies, exhibiting an 8-way embarrassingly parallel pattern [4].”
BackgroundMethodInfluentialSelf-citationGraph of Thoughts: Solving Elaborate Problems with Large Language Models
Besta, M., Blach, N., Kubicek, A., Gerstenberger, R., Podstawski, M., Gianinazzi, L., Gajda, J., Lehmann, T., Niewiadomski, H., Nyczyk, P., et al. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i16.29720 · Cited by 556 · cited 1× · see citing works
“…of Large Language Models (LLMs) have expanded the applicability of LLMs beyond content generation to solving complex problems [51, 23, 57, 5, 50, 63, 8, 55, 12]; and recent works have shown how this reasoning capability could be helpful in improving accuracy for solving complex and…”
BackgroundProgram Synthesis with Large Language Models
Austin, J., Odena, A., Nye, M., Bosma, M., Michalewski, H., Dohan, D., Jiang, E., Cai, C., Terry, M., Le, Q., et al. (2021) · arXiv · doi:10.48550/arxiv.2108.07732 · Cited by 388 · cited 1× · see citing works
“This limits its applicability in complex scenarios such as coding [7, 33, 14, 3] or math [16, 15] problems, as also stated in the paper [34].”
BackgroundLeast-to-Most Prompting Enables Complex Reasoning in Large Language Models
Zhou, D., Schärli, N., Hou, L., Wei, J., Scales, N., Wang, X., Schuurmans, D., Cui, C., Bousquet, O., Le, Q., et al. (2022) · arXiv · doi:10.48550/arxiv.2205.10625 · Cited by 380 · cited 2× · see citing works
“…of Large Language Models (LLMs) have expanded the applicability of LLMs beyond content generation to solving complex problems [51, 23, 57, 5, 50, 63, 8, 55, 12]; and recent works have shown how this reasoning capability could be helpful in improving accuracy for solving complex and logical…”
BackgroundMeasuring Mathematical Problem Solving With the MATH Dataset
Hendrycks, D., Burns, C., Kadavath, S., Arora, A., Basart, S., Tang, E., Song, D., & Steinhardt, J. (2021) · arXiv · doi:10.48550/arxiv.2103.03874 · Cited by 306 · cited 1× · see citing works
“This limits its applicability in complex scenarios such as coding [7, 33, 14, 3] or math [16, 15] problems, as also stated in the paper [34].”
BackgroundMeasuring and Narrowing the Compositionality Gap in Language Models
Press, O., Zhang, M., Min, S., Schmidt, L., Smith, N. A., & Lewis, M. (2022) · arXiv · doi:10.48550/arxiv.2210.03350 · Cited by 294 · cited 1× · see citing works
“Several studies [52, 38, 40, 63, 13] have explored prompting methods of breaking down complex queries into various levels of detail to solve them, thereby improving LLM’s performance in reasoning tasks.”
BackgroundMeasuring Massive Multitask Language Understanding
Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., & Steinhardt, J. (2020) · arXiv · doi:10.48550/arxiv.2009.03300 · Cited by 291 · cited 1× · see citing works
“LLMCompiler addresses this by translating an input query into a series of tasks with inter-dependencies, thereby expanding the spectrum of problems it can handle.”
BackgroundTaskMatrix.AI: Completing Tasks by Connecting Foundation Models with Millions of APIsPreprint
Liang, Y., Wu, C., Song, T., Wu, W., Xia, Y., Liu, Y., Ou, Y., Lu, S., Ji, L., Mao, S., et al. (2023) · Intelligent Computing · doi:10.48550/arxiv.2303.16434 · Cited by 286 · cited 1× · see citing works
“Below, we provide the Planner example prompts that are used to set up the framework for the Movie Recommendation and Game of 24 benchmarks with only a few lines of prompts.”
Take a Step Back: Evoking Reasoning via Abstraction in Large Language ModelsPreprint
Zheng, H. S., Mishra, S., Chen, X.-Y., Cheng, H.-T., Chi, E. H., Le, Q. V., & Zhou, D. (2023) · International Conference on Learning Representations · doi:10.48550/arxiv.2310.06117 · Cited by 263 · cited 1× · see citing works
“Step-Back Prompting [61] enables LLMs to abstract high-level concepts from details to enhance reasoning abilities across various tasks.”
BackgroundBreak It Down: A Question Understanding Benchmark
Wolfson, T., Geva, M., Gupta, A., Gardner, M., Goldberg, Y., Deutch, D., & Berant, J. (2020) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00309 · Cited by 241 · cited 1× · see citing works
“Several studies [52, 38, 40, 63, 13] have explored prompting methods of breaking down complex queries into various levels of detail to solve them, thereby improving LLM’s performance in reasoning tasks.”
BackgroundGorilla: Large Language Model Connected with Massive APIs
Patil, S., Zhang, T., Wang, X., & Gonzalez, J. (2024) · Advances in Neural Information Processing Systems 37 · doi:10.52202/079017-4020 · Cited by 226 · cited 1× · see citing works
“Gorilla [39] introduced a finetuned LLM designed for function calling, and ToolLLM [41] and RestGPT [46] have extended LLMs to support real-world APIs.”
BackgroundPAL: Program-aided Language Models
Gao, L., Madaan, A., Zhou, S., Alon, U., Liu, P., Yang, Y., Callan, J., & Neubig, G. (2022) · arXiv · doi:10.48550/arxiv.2211.10435 · Cited by 155 · cited 1× · see citing works
“…of Large Language Models (LLMs) have expanded the applicability of LLMs beyond content generation to solving complex problems [51, 23, 57, 5, 50, 63, 8, 55, 12]; and recent works have shown how this reasoning capability could be helpful in improving accuracy for solving complex and logical tasks.”
BackgroundMeasuring Coding Challenge Competence With APPS
Hendrycks, D., Basart, S., Kadavath, S., Mazeika, M., Arora, A., Guo, E., Burns, C., Puranik, S., He, H., Song, D., et al. (2021) · arXiv · doi:10.48550/arxiv.2105.09938 · Cited by 142 · cited 1× · see citing works
“This limits its applicability in complex scenarios such as coding [7, 33, 14, 3] or math [16, 15] problems, as also stated in the paper [34].”
BackgroundSparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot
Frantar, E., & Alistarh, D. (2023) · arXiv · doi:10.48550/arxiv.2301.00774 · Cited by 131 · cited 1× · see citing works
“Various studies have focused on optimizing model design [21, 11, 30, 9, 24, 10, 22, 6, 28] and systems [25, 59, 1, 2] for efficient LLM inference.”
BackgroundChen, W., Ma, X., Wang, X., & Cohen, W. W. (2022) · arXiv · doi:10.48550/arxiv.2211.12588 · Cited by 125 · cited 1× · see citing works
“…of Large Language Models (LLMs) have expanded the applicability of LLMs beyond content generation to solving complex problems [51, 23, 57, 5, 50, 63, 8, 55, 12]; and recent works have shown how this reasoning capability could be helpful in improving accuracy for solving complex and logical tasks.”
BackgroundAPI-Bank: A Comprehensive Benchmark for Tool-Augmented LLMs
Li, M., Zhao, Y., Yu, B., Song, F., Li, H., Yu, H., Li, Z., Huang, F., & Li, Y. (2023) · arXiv · doi:10.48550/arxiv.2304.08244 · Cited by 124 · cited 1× · see citing works
“Second, LLMCompiler supports dynamic replanning which is important for problems whose execution flow cannot be determined statically in the beginning (Sec.”
ALFWorld: Aligning Text and Embodied Environments for Interactive Learning
Shridhar, M., Yuan, X., Côté, M.-A., Bisk, Y., Trischler, A., & Hausknecht, M. (2020) · arXiv · doi:10.48550/arxiv.2010.03768 · Cited by 80 · cited 1× · see citing works
“A notable work is ReWOO [53] which employs a planner to separate the reasoning process from the execution and observation phases to decrease token usage and cost as compared to ReAct.”
MethodLASER: LLM Agent with State-Space Exploration for Web Navigation
Ma, K., Zhang, H., Wang, H., Pan, X., & Yu, D. (2023) · arXiv.org · doi:10.48550/arxiv.2309.08172 · Cited by 72
Qin, Y., Liang, S., Ye, Y., Zhu, K., Yan, L., Lu, Y., Lin, Y., Cong, X., Tang, X., Qian, B., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2307.16789 · Cited by 71 · cited 2× · see citing works
“This approach, along with the agent systems that extend ReAct [20, 57, 41, 42, 47], may lead to inefficiencies in latency and cost due to the sequential function calling and repetitive LLM invocations for each reasoning and action step.”
BackgroundFast Inference from Transformers via Speculative Decoding
Leviathan, Y., Kalman, M., & Matias, Y. (2022) · arXiv · doi:10.48550/arxiv.2211.17192 · Cited by 71 · cited 1× · see citing works
“Various studies have focused on optimizing model design [21, 11, 30, 9, 24, 10, 22, 6, 28] and systems [25, 59, 1, 2] for efficient LLM inference.”
BackgroundGPT-NeoX-20B: An Open-Source Autoregressive Language Model
Black, S., Biderman, S., Hallahan, E., Anthony, Q., Gao, L., Golding, L., He, H., Leahy, C., McDonell, K., Phang, J., et al. (2022) · arXiv · doi:10.48550/arxiv.2204.06745 · Cited by 67 · cited 1× · see citing works
“LLMCompiler enables efficient parallel function calling for open-source models, and also, as we will show later in Sec.”
MethodIs a Question Decomposition Unit All We Need?Preprint
Patel, P. H., Mishra, S., Parmar, M., & Baral, C. (2022) · Conference on Empirical Methods in Natural Language Processing · doi:10.48550/arxiv.2205.12538 · Cited by 67 · cited 1× · see citing works
“Several studies [52, 38, 40, 63, 13] have explored prompting methods of breaking down complex queries into various levels of detail to solve them, thereby improving LLM’s performance in reasoning tasks.”
BackgroundA Fast Post-Training Pruning Framework for Transformers
Kwon, W., Kim, S., Mahoney, M. W., Hassoun, J., Keutzer, K., & Gholami, A. (2022) · arXiv · doi:10.48550/arxiv.2204.09656 · Cited by 64 · cited 1× · see citing works
“Various studies have focused on optimizing model design [21, 11, 30, 9, 24, 10, 22, 6, 28] and systems [25, 59, 1, 2] for efficient LLM inference.”
BackgroundSelf-citationAccelerating Large Language Model Decoding with Speculative Sampling
Chen, C., Borgeaud, S., Irving, G., Lespiau, J.-B., Sifre, L., & Jumper, J. (2023) · arXiv · doi:10.48550/arxiv.2302.01318 · Cited by 40 · cited 1× · see citing works
“Various studies have focused on optimizing model design [21, 11, 30, 9, 24, 10, 22, 6, 28] and systems [25, 59, 1, 2] for efficient LLM inference.”
BackgroundIfQA: A Dataset for Open-domain Question Answering under Counterfactual PresuppositionsPreprint
Yu, W., Jiang, M., Clark, P., & Sabharwal, A. (2023) · Conference on Empirical Methods in Natural Language Processing · doi:10.48550/arxiv.2305.14010 · Cited by 38 · cited 1× · see citing works
“Inspired by the IfQA benchmark [60], ParallelQA contains 113 examples that involve mathematical questions on factual attributes of various entities.”
MethodSkeleton-of-Thought: Prompting LLMs for Efficient Parallel Generation
Ning, X., Lin, Z., Zhou, Z., Wang, Z., Yang, H., & Wang, Y. (2023) · arXiv · doi:10.48550/arxiv.2307.15337 · Cited by 36 · cited 2× · see citing works
“Skeleton-of-Thought [34] recently proposed to reduce latency through application-level parallel decoding.”
BackgroundIdentifying the Risks of LM Agents with an LM-Emulated Sandbox
Ruan, Y., Dong, H., Wang, A., Pitis, S., Zhou, Y., Ba, J., Dubois, Y., Maddison, C. J., & Hashimoto, T. (2023) · arXiv · doi:10.48550/arxiv.2309.15817 · Cited by 19 · cited 1× · see citing works
“This approach, along with the agent systems that extend ReAct [20, 57, 41, 42, 47], may lead to inefficiencies in latency and cost due to the sequential function calling and repetitive LLM invocations for each reasoning and action step.”
BackgroundReWOO: Decoupling Reasoning from Observations for Efficient Augmented Language Models
Xu, B., Peng, Z., Lei, B., Mukherjee, S., Liu, Y., & Xu, D. (2023) · arXiv · doi:10.48550/arxiv.2305.18323 · Cited by 17 · cited 2× · see citing works
“A notable work is ReWOO [53] which employs a planner to separate the reasoning process from the execution and observation phases to decrease token usage and cost as compared to ReAct.”
BackgroundMethodTPTU: Large Language Model-based AI Agents for Task Planning and Tool Usage
Ruan, J., Chen, Y., Zhang, B., Xu, Z., Bao, T., Du, G., Shi, S., Mao, H., Li, Z., Zeng, X., et al. (2023) · arXiv · doi:10.48550/arxiv.2308.03427 · Cited by 11 · see citing works
Flagged first, then by citations
10 of the 56 references couldn’t be matched to a known paper.
Research works printed but not linked: 2
- Frantar, E., Ashkboos, S., Hoefler, T., and Alistarh, D. GPTQ: Accurate post-training compression for generative pretrained transformers. arXiv preprint arXiv:2210.17323, 2022. · arXiv
- Yu, G.-I., Jeong, J. S., Kim, G.-W., Kim, S., and Chun, B.- G. Orca: A distributed serving system for {Transformer- Based} generative models. In 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22), pp. 521-538, 2022.
Cited by
Sorted by citations
RATT: A Thought Structure for Coherent and Correct LLM Reasoning
Zhang, J., Wang, X., Ren, W., Lu, J., Wang, D., & Liu, K. (2025) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v39i25.34876 · Cited by 43 · see citing works
TypeFly: Low-Latency Drone Planning With Large Language Models
Chen, G., Yu, X., Ling, N., & Zhong, L. (2025) · IEEE Transactions on Mobile Computing · doi:10.1109/tmc.2025.3561282 · Cited by 31 · see citing works
“Related to TypeFly’s support of stream interpretation of Min-iSpec plans, the LLM Compiler [17] asynchronously executes functions and generates programs by analyzing dependencies between variables.”
KG.GOV: Knowledge graphs as the backbone of data governance in AI
Meroño-Peñuela, A., Simperl, E. P. B., Kurteva, A., & Reklos, I. (2024) · Journal of Web Semantics · doi:10.1016/j.websem.2024.100847 · Cited by 27 · see citing works
TinyAgent: Function Calling at the Edge
Erdogan, L. E., Lee, N., Jha, S., Kim, S., Tabrizi, R., Moon, S., Hooper, C. R. C., Anumanchipalli, G. K., Keutzer, K., & Gholami, A. (2024) · Conference on Empirical Methods in Natural Language Processing: System Demonstrations (EMNLP) · doi:10.18653/v1/2024.emnlp-demo.9 · Cited by 21 · see citing works
Self-citationOrchestrating Agents and Data for Enterprise: A Blueprint Architecture for Compound AI
Kandogan, E., Bhutani, N., Zhang, D., Chen, R. L., Gurajada, S., & Hruschka, E. (2025) · 2025 IEEE 41st International Conference on Data Engineering Workshops (ICDEW) · doi:10.1109/icdew67478.2025.00007 · Cited by 16 · see citing works
“Despite emerging attempts to explore LLMs’ reasoning capabilities and use them as planners [11], [58] or ‘routers’ of existing tools and APIs [17], [23], [35], LLMs alone still cannot solve the planning problem [13], [42].”
Towards End-to-End Optimization of LLM-based Applications with Ayo
Tan, X., Jiang, Y., Yang, Y., & Xu, H. (2025) · Proceedings of the 30th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2 · doi:10.1145/3676641.3716278 · Cited by 13 · see citing works
“To mitigate such problems, many techniques have been proposed, including RAG (Retrieval Augmented Generation) [49, 57], external function calls [16, 41, 47] and even multiple LLM interactions.”
Dimitropoulos, N., Kaipis, M., Giartzas, S., & Michalos, G. (2025) · CIRP Annals · doi:10.1016/j.cirp.2025.04.081 · Cited by 13 · see citing works
SGLang: Efficient Execution of Structured Language Model Programs
Zheng, L., Yin, L., Xie, Z., Sun, C., Huang, J., Yu, C. H., Cao, S., Kozyrakis, C., Stoica, I., Gonzalez, J. E., et al. (2023) · arXiv · doi:10.48550/arxiv.2312.07104 · Cited by 145 · see citing works
“Several LLM programming and agent frameworks exist, such as Guidance [13], LMQL [4], DSPy [20], LangChain [24], AutoGen [55], and LLM Compiler [21].”
Less is More: Optimizing Function Calling for LLM Execution on Edge Devices
Paramanayakam, V., Καρατζάς, Α., Anagnostopoulos, I., & Stamoulis, D. S. (2025) · Design, Automation, and Test in Europe · doi:10.23919/date64628.2025.10992798 · Cited by 11 · see citing works
“To handle LLM errors at runtime, we employ a “fallback” mechanism, as is common in LLM “compilers” [29].”
RAG-Optimized Tibetan Tourism LLMs: Enhancing Accuracy and Personalization
Qi, J., Yan, S., Zhang, Y., Zhang, W., Jin, R., Hu, Y., & Wang, K. (2024) · doi:10.1145/3703935.3704112 · Cited by 11 · see citing works
“After identifying the viewpoints based on user needs, the geographical and historical information from the knowledge source database was returned to the large language model as an external knowledge base.”
Cinquin, O. (2024) · Briefings in Bioinformatics · doi:10.1093/bib/bbaf045 · Cited by 10 · see citing works
Function Calling in Large Language Models: Industrial Practices, Challenges, and Future Directions
Wang, M., Zhang, Y., Yu, B., Hao, B., Peng, C., Chen, Y.-C., Zhou, W., Gu, J., Zhuang, C., Guo, R., et al. (2026) · ACM Computing Surveys · doi:10.1145/3788284 · Cited by 10 · see citing works
“LLM Compiler [52, 132] effectively plans and executes calls in parallel using a Directed Acyclic Graph (DAG) to manage dependencies, significantly reducing latency in complex scenarios and improving throughput.”
DrAgent: An Agentic Approach to Fault Analysis in Power Grids Using Large Language Models
Saha, B. K., Aarthi, V., & Naidu, O. D. (2025) · International Conference on Artificial Intelligence in Information and Communication (ICAIIC) · doi:10.1109/icaiic64266.2025.10920654 · Cited by 7 · see citing works
“LLMCompiler [13], on the other hand, offers further improvement by enabling tool execution in parallel, thereby potentially reducing the response time.”
Divide-Then-Aggregate: An Efficient Tool Learning Method via Parallel Tool Invocation
Zhu, D., Shi, W., Shi, Z., Ren, Z., Wang, S., Yan, L., & Yin, D. (2025) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2025.acl-long.1401 · Cited by 7 · see citing works
Budget-Constrained Tool Learning with Planning
Zheng, Y., Li, P., Yan, M., Zhang, J., Huang, F., & Liu, Y. (2024) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2024.findings-acl.536 · Cited by 6 · see citing works
(2024) · doi:10.1145/3642970.3655844 · Cited by 4 · see citing works
LEGOMem: Modular Procedural Memory for Multi-agent LLM Systems for Workflow Automation
Han, D., Couturier, C., Madrigal Diaz, D., Zhang, X., Rühle, V., & Rajmohan, S. (2026) · Proceedings of the 25th International Conference on Autonomous Agents and Multiagent Systems · doi:10.65109/vlua1303 · Cited by 3 · see citing works
“Large Language Models (LLMs) are increasingly deployed as agents to automate complex multi-step workflows [2, 3, 5, 13, 19, 20, 22, 25– 28, 33, 35, 39].”
CMDBench: A Benchmark for Coarse-to-fine Multimodal Data Discovery in Compound AI Systems
Feng, Y., Rahman, S., Feng, A., Chen, V. S., & Kandogan, E. (2024) · doi:10.1145/3665601.3669846 · Cited by 3 · see citing works
“However, as mentioned earlier, these collections can be grouped to formulate additional collections on dimensions such as teams-specific collections and corresponding separate databases.”
BackgroundSelf-Evolving Programs: A Novel Approach Leveraging LLMs and Quine Programs
Saghiri, A. M., & Wang, N. (2024) · International Conference on Computing, Internet of Things and Microwave Systems (ICCIMS) · doi:10.1109/iccims61672.2024.10690672 · Cited by 2 · see citing works
“This shift sets the stage for our exploration of new, more flexible methodologies [3], [9], [10].”
BackgroundSimple Action Model: Enabling LLM to Sequential Function Calling Tool Chain
Sen, R. S., Amalkrishna, M., Prithviraj, R., Jose, S. P., Joseph, N., & Thomas, R. (2024) · International Conference on Advancement in Renewable Energy and Intelligent Systems (AREIS) · doi:10.1109/areis62559.2024.10893677 · Cited by 2 · see citing works
Multi-Agent Chatbot and Its Practical Applications
Dhait, S., Sapate, A., Deshmukh, P., Gupta, G. K., Singh, A. K., & Kishor, R. (2026) · Lecture notes in electrical engineering · doi:10.1007/978-981-96-7241-7_21 · Cited by 2 · see citing works
Challagundla, B. C., & B, G. R. (2024) · Research Square · doi:10.21203/rs.3.rs-4354348/v1 · Cited by 2 · see citing works
Dong, H., Su, Q., Gao, Y., Li, Z., Ruan, Y., Pekhimenko, G. G., Maddison, C. J., & Si, X. (2025) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2025.acl-long.63 · Cited by 1 · see citing works
Multi-Modal Data Exploration via Language Agents
Nooralahzadeh, F., Zhang, Y., Fürst, J., & Stockinger, K. (2025) · International Joint Conference on Natural Language Processing and Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (IJCNLP-AACL) · doi:10.18653/v1/2025.findings-ijcnlp.47 · Cited by 1 · see citing works
“M 2 EX is an agentic system (Kapoor et al., 2024) driven by LLMCom-piler (Kim et al., 2023), a dynamic planner pattern based on a Large Language Model, equipped with a comprehensive toolkit T containing all the necessary models to decompose a user’s request, such as a multi-modal natural language question, into a work…”
Continuum: Efficient and Robust Multi-Turn LLM Agent Scheduling with KV Cache Time-to-Live
Li, H., He, R., Mang, Q., Zhang, Q., Mao, H., Chen, X., Zhou, H., Zhang, H., Cheung, A., Gonzalez, J., et al. (2025) · arXiv · doi:10.48550/arxiv.2511.02230
“Since agentic are mostly post-trained beforehand, Continuum can mitigate this by using the distribution during training for handling cold start.”
CRALA: A Chinese-centric Risk Simulation and Assessment Framework for LLM Agents
Ye, W., Zhang, Z., Li, W., Qin, Y., & Zhang, P. (2024) · International Symposium on Artificial Intelligence and Intelligent Manufacturing (AIIM) · doi:10.1109/aiim64537.2024.10934441
“For example, the ChatGPT plugin and LLM-based mobile control interfaces [4,5] are both good at understanding complex user instructions and completing a wide range of tasks [6].”
Integração de LLMs com APIs : um estudo de caso
Silva, L. L., & Ponciano, J. R. (2024) · doi:10.11606/003226610
PlanCompiler: A Deterministic Compilation Architecture for Structured Multi-Step LLM PipelinesPreprint
Harikumar, P. (2026) · Research Square · doi:10.21203/rs.3.rs-9186427/v1
D3X: Dependency Driven, Decentralised Execution for Scalable AI Teams
Rawlot, G. S., Ouarbya, S., & Ouarbya, L. (2025) · International Conference on Advances in Artificial Intelligence (ICAAI) · doi:10.1145/3787279.3787317
TimelyLLM: Time-sensitive LLM Serving System for Physical-I/O Limited Agents
Ling, N., Chen, G., Khandelwal, A., & Zhong, L. (2026) · Proceedings of the 24th Annual International Conference on Mobile Systems, Applications and Services · doi:10.1145/3745756.3809203
“Another line of research enables the asynchronous execution of LLM-generated plans [11, 25], streaming outputs directly to executors.”
The Agent Loop: A Survey of Control Strategies, Skills, and Harnesses for LLM AgentsPreprint
Lee, J., & Park, C. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.7186738
Wohlschläger, L., Falkenreck, A., & Fottner, J. (2026) · Circular Economy · doi:10.55845/hv5ywx12
“The autonomous use of selected tools enables agents to overcome the typical weaknesses of LLMs, such as knowledge gaps, computational errors, or the ability to utilize real-time data (Kim et al., 2024; Sapkota et al., 2026).”
Носко, С. В., Hulevych, M., Коломійцев, О., Третяк, В. Ф., Галицький, О. Ф., Гайбадулов, Б. В., Моргун, Є. В., Клєвцов, С., Корольков, К., Бєлоус, М., et al. (2026) · InterConf · doi:10.51582/interconf.19-20.10.2026.026
- Authors
- Kim, S., Moon, S., Tabrizi, R., Lee, N., Mahoney, M. W., Keutzer, K., & Gholami, A.
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2023
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2312.04511
- OpenAlex
- W4389500120
- Cited by
- 12
- References
- 56
- Citations by year
- 2025: 6 · 2024: 6
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Science Foundation
- U.S. Department of Energy
- Intel Corporation
- Samsung Advanced Institute of Technology
- Korea Foundation for Advanced Studies
- Samsung
- Intelligence Advanced Research Projects Activity
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Artificial Intelligence Applications
- Keywords
- task dispatching, cost reduction, LLM reasoning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
