LLM-Based Agents for Tool Learning: A Survey
The DOI matches its Crossref record; 52 of the 52 references it cites were found in a registry or an index.
Abstract
Human beings capable of making and using tools can accomplish tasks far beyond their innate abilities, and this paradigm of integration with tools may not be limited to humans themselves. Recently, the large language model (LLM) has demonstrated immense potential across various fields with its unique planning and reasoning abilities. However, there are still many challenges beyond its capabilities due to deficiencies in its training data and inherent illusions. Thus, integrating LLMs and tools into tool learning agents has become a new emerging research direction. To this end, we present a systematic investigation and comprehensive review of tool-learning agents in this paper. We start by introducing the definition of the tool learning task for Agents and then illustrating the typical architecture of the tool-learning models. Since these tools are all defined by users, LLM does not know what tools there are and what their functions are. Thus, LLMs should first find appropriate tools and split the tool retrieval methods into two categories: training-based and non-training-based. To accurately complete the user task, it is important to decompose the task into several sub-tasks and execute them in the correct order. Following that, we introduce the tool planning methods and organize these works by whether they rely on the model’s inherent reasoning capabilities for planning or utilize external reasoning tools. Due to the rapid development of this field, we also introduce an emerging frontier direction: using multimodal tools for LLM. In addition, we compile current open-source benchmarks and evaluation metrics, focusing on their scale, composition, calculation methods, and assessment dimensions. Next, we introduce several application scenarios for the LLM-based tool learning methods. Finally, we discuss the safety and ethical issues involved in tool learning.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Billion-scale similarity search with GPUs
Johnson, J., Douze, M., & Jegou, H. (2021) · IEEE Transactions on Big Data · doi:10.1109/tbdata.2019.2921572 · Cited by 4,228
Billion-scale similarity search with GPUs
Johnson, J., Douze, M., & Jégou, H. (2017) · arXiv · doi:10.48550/arxiv.1702.08734 · Cited by 4,140 · cited 1×
“Upon receiving feedback from the environment, LLM processes the information, formalizes it into PDDL language, and then employs a BFS [60] solver to generate a plan.”
A Survey of Monte Carlo Tree Search Methods
Browne, C. B., Powley, E., Whitehouse, D., Lucas, S. M., Cowling, P. I., Rohlfshagen, P., Tavener, S., Perez, D., Samothrakis, S., & Colton, S. (2012) · IEEE Transactions on Computational Intelligence and AI in Games · doi:10.1109/tciaig.2012.2186810 · Cited by 2,970 · cited 1×
“Building (9) P = p 1 , p 2 , ⋯ , p n = plan( LLM , g ; Θ , T ) ; on this, ToT employs the MCTS algorithm [11] to identify the optimal planning path.”
Generative Agents: Interactive Simulacra of Human Behavior
Park, J. S., O'Brien, J., Cai, C. J., Morris, M. R., Liang, P., & Bernstein, M. S. (2023) · Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology · doi:10.1145/3586183.3606763 · Cited by 2,187
Generative Agents: Interactive Simulacra of Human Behavior
Park, J. S., O'Brien, J. C., Cai, C. J., Morris, M. R., Liang, P., & Bernstein, M. S. (2023) · arXiv · doi:10.48550/arxiv.2304.03442 · Cited by 2,173
Natural Questions: A benchmark for question answering research
Kwiatkowski, T., Palomaki, J., Redfield, O., Collins, M., Parikh, A., Alberti, C., Epstein, D., Polosukhin, I., Devlin, J., Lee, K., et al. (2019) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00276 · Cited by 2,069
HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering
Yang, Z., Qi, P., Zhang, S., Bengio, Y., Cohen, W., Salakhutdinov, R., & Manning, C. D. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1259 · Cited by 1,804 · cited 1×
“Some people also evaluate the finetuned models on other non-tool-specific general datasets, which require multi-step reasoning, such as mathematical reasoning [23, 87, 97], and Wikipedia(WikiQA) [65, 142].”
The imperative for regulatory oversight of large language models (or generative AI) in healthcare
Meskó, B., & Topol, E. J. (2023) · npj Digital Medicine · doi:10.1038/s41746-023-00873-0 · Cited by 1,047
ImageBind One Embedding Space to Bind Them All
Girdhar, R., El-Nouby, A., Liu, Z., Singh, M., Alwala, K. V., Joulin, A., & Misra, I. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.01457 · Cited by 863 · cited 1×
“Specifically, it adopts ImageBind [38] as the primary multimodal encoder, enabling the system to align and interpret inputs across various modalities effectively.”
Graph of Thoughts: Solving Elaborate Problems with Large Language Models
Besta, M., Blach, N., Kubicek, A., Gerstenberger, R., Podstawski, M., Gianinazzi, L., Gajda, J., Lehmann, T., Niewiadomski, H., Nyczyk, P., et al. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i16.29720 · Cited by 556
Graph of Thoughts: Solving Elaborate Problems with Large Language Models
Besta, M., Blach, N., Kubicek, A., Gerstenberger, R., Podstawski, M., Gianinazzi, L., Gajda, J., Lehmann, T., Niewiadomski, H., Nyczyk, P., et al. (2023) · arXiv · doi:10.48550/arxiv.2308.09687 · Cited by 556
Using an LLM to Help With Code Understanding
Nam, D., Macvean, A., Hellendoorn, V., Vasilescu, B., & Myers, B. (2024) · Proceedings of the IEEE/ACM 46th International Conference on Software Engineering · doi:10.1145/3597503.3639187 · Cited by 487 · cited 3×
“For code understanding, an IDE plugin [92] enhances code comprehension by integrating API interpreters directly within the IDE compiler.”
Rico: A Mobile App Dataset for Building Data-Driven Design Applications
Deka, B., Huang, Z., Franzen, C., Hibschman, J., Afergan, D., Li, Y., Nichols, J., & Kumar, R. (2017) · Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology · doi:10.1145/3126594.3126651 · Cited by 482
VirtualHome: Simulating Household Activities Via Programs
Puig, X., Ra, K., Boben, M., Li, J., Wang, T., Fidler, S., & Torralba, A. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00886 · Cited by 378
Creating Large Language Model Applications Utilizing LangChain: A Primer on Developing LLM Apps Fast
Topsakal, O., & Akinci, T. C. (2023) · International Conference on Applied Engineering and Natural Sciences · doi:10.59287/icaens.1127 · Cited by 331 · cited 1×
“Pedro et al. [99] has reduced the vulnerability of databases to injection attacks by augmenting the Langchain framework [121] with tool extensions.”
Are NLP Models really able to Solve Simple Math Word Problems?
Patel, A., Bhattamishra, S., & Goyal, N. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2021.naacl-main.168 · Cited by 307
MM-LLMs: Recent Advances in MultiModal Large Language Models
Zhang, D., Yu, Y., Dong, J., Li, C., Su, D., Chu, C., & Yu, D. (2024) · Findings of the Association for Computational Linguistics ACL 2024 · doi:10.18653/v1/2024.findings-acl.738 · Cited by 260
CogAgent: A Visual Language Model for GUI Agents
Hong, W., Wang, W., Lv, Q., Xu, J., Yu, W., Ji, J., Wang, Y., Wang, Z., Dong, Y., Ding, M., et al. (2024) · 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52733.2024.01354 · Cited by 202 · cited 1×
“CogAgent [48] introduces a cognitively-inspired approach, encoding UI structures and action semantics into a shared embedding space to improve alignment between intent and interface.”
Tool Learning with Foundation Models
Qin, Y., Hu, S., Lin, Y., Chen, W., Ding, N., Cui, G., Zeng, Z., Huang, Y., Xiao, C., Han, C., et al. (2023) · arXiv · doi:10.48550/arxiv.2304.08354 · Cited by 184
(2024) · doi:10.1145/3704435 · Cited by 184
Jin, Q., Yang, Y., Chen, Q., & Lu, Z. (2024) · Bioinformatics · doi:10.1093/bioinformatics/btae075 · Cited by 176 · cited 2×
“Federated Learning Under the premise [58, 133] that advanced LLMs are unwilling to open-source their training data, various organizations building their tool learning agents and collaborating represent a mutually beneficial and win-win model.”
SciREX: A Challenge Dataset for Document-Level Information Extraction
Jain, S., van Zuylen, M., Hajishirzi, H., & Beltagy, I. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.670 · Cited by 159
Enabling Conversational Interaction with Mobile UI using Large Language Models
Wang, B., Li, G., & Li, Y. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3580895 · Cited by 156
InjecAgent: Benchmarking Indirect Prompt Injections in Tool-Integrated Large Language Model Agents
Zhan, Q., Liang, Z., Ying, Z., & Kang, D. (2024) · Findings of the Association for Computational Linguistics: ACL 2024 · doi:10.18653/v1/2024.findings-acl.624 · Cited by 147
Logic-LM: Empowering Large Language Models with Symbolic Solvers for Faithful Logical Reasoning
Pan, L., Albalak, A., Wang, X., & Wang, W. Y. (2023) · arXiv · doi:10.48550/arxiv.2305.12295 · Cited by 144 · cited 1×
“Similarly, Logic-lm [94] transforms described problems into first-order formal logic (FOL [7]).”
InjecAgent: Benchmarking Indirect Prompt Injections in Tool-Integrated Large Language Model Agents
Zhan, Q., Liang, Z., Ying, Z., & Kang, D. (2024) · arXiv · doi:10.48550/arxiv.2403.02691 · Cited by 143
Android in the Wild: A Large-Scale Dataset for Android Device Control
Rawles, C., Li, A., Rodriguez, D., Riva, O., & Lillicrap, T. (2023) · arXiv · doi:10.48550/arxiv.2307.10088 · Cited by 141 · cited 1×
“To address this, Auto-UI [159] leverages more realistic and diverse data sources, notably from Android in the Wild (AitW) [104], and constructs a collection of routine app testing tasks via crowdsourced annotation.”
A Diverse Corpus for Evaluating and Developing English Math Word Problem Solvers
Miao, S.-Y., Liang, C.-C., & Su, K.-Y. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.92 · Cited by 135
API-Bank: A Comprehensive Benchmark for Tool-Augmented LLMs
Li, M., Zhao, Y., Yu, B., Song, F., Li, H., Yu, H., Li, Z., Huang, F., & Li, Y. (2023) · arXiv · doi:10.48550/arxiv.2304.08244 · Cited by 124
API-Bank: A Comprehensive Benchmark for Tool-Augmented LLMs
Li, M., Zhao, Y., Yu, B., Song, F., Li, H., Yu, H., Li, Z., Huang, F., & Li, Y. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.187 · Cited by 124 · cited 1×
“API-Bank [67] is one of the earliest tool-learning benchmarks, classifying tasks into single-API or multi-API calls.”
Ni, B., & Buehler, M. J. (2024) · Extreme Mechanics Letters · doi:10.1016/j.eml.2024.102131 · Cited by 113
LLM+P: Empowering Large Language Models with Optimal Planning Proficiency
Liu, B., Jiang, Y., Zhang, X., Liu, Q., Zhang, S., Biswas, J., & Stone, P. (2023) · arXiv · doi:10.48550/arxiv.2304.11477 · Cited by 104
Video PreTraining (VPT): Learning to Act by Watching Unlabeled Online Videos
Baker, B., Akkaya, I., Zhokhov, P., Huizinga, J., Tang, J., Ecoffet, A., Houghton, B., Sampedro, R., & Clune, J. (2022) · arXiv · doi:10.48550/arxiv.2206.11795 · Cited by 75
A Multimodal Foundation Agent for Financial Trading: Tool-Augmented, Diversified, and Generalist
Zhang, W., Zhao, L., Xia, H., Sun, S., Sun, J., Qin, M., Li, X., Zhao, Y., Zhao, Y., Cai, X., et al. (2024) · Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining · doi:10.1145/3637528.3671801 · Cited by 74
Shi, W., Xu, R., Zhuang, Y., Yu, Y., Zhang, J., Wu, H., Zhu, Y., Ho, J. C., Yang, C., & Wang, M. D. (2024) · Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2024.emnlp-main.1245 · Cited by 54
SwiftSage: A Generative Agent with Fast and Slow Thinking for Complex Interactive Tasks
Lin, B. Y., Fu, Y., Yang, K., Brahman, F., Huang, S., Bhagavatula, C., Ammanabrolu, P., Choi, Y., & Ren, X. (2023) · Advances in Neural Information Processing Systems 36 · doi:10.52202/075280-1034 · Cited by 50
R-Judge: Benchmarking Safety Risk Awareness for LLM Agents
Yuan, T., He, Z., Dong, L., Wang, Y., Zhao, R., Xia, T., Xu, L., Zhou, B., Li, F., Zhang, Z., et al. (2024) · Findings of the Association for Computational Linguistics: EMNLP 2024 · doi:10.18653/v1/2024.findings-emnlp.79 · Cited by 45
Auto-GPT for Online Decision Making: Benchmarks and Additional Opinions
Yang, H., Yue, S., & He, Y. (2023) · arXiv · doi:10.48550/arxiv.2306.02224 · Cited by 44
Ferret-UI: Grounded Mobile UI Understanding with Multimodal LLMs
You, K., Zhang, H., Schoop, E., Weers, F., Swearngin, A., Nichols, J., Yang, Y., & Gan, Z. (2024) · arXiv · doi:10.48550/arxiv.2404.05719 · Cited by 40 · cited 1×
“Ferret-UI [151] extends the Ferret framework to GUI scenarios, supporting visual grounding through pixel-wise element understanding and action prediction.”
Coupling Large Language Models with Logic Programming for Robust and General Reasoning from Text
Yang, Z., Ishay, A., & Lee, J. (2023) · Findings of the Association for Computational Linguistics: ACL 2023 · doi:10.18653/v1/2023.findings-acl.321 · Cited by 38
TALM: Tool Augmented Language Models
Parisi, A., Zhao, Y., & Fiedel, N. (2022) · arXiv · doi:10.48550/arxiv.2205.12255 · Cited by 35
LLM Multi-Agent Systems: Challenges and Open Problems
Han, S., Zhang, Q., Jin, W., & Xu, Z. (2024) · arXiv · doi:10.48550/arxiv.2402.03578 · Cited by 33
T-Eval: Evaluating the Tool Utilization Capability of Large Language Models Step by Step
Chen, Z., Du, W., Zhang, W., Liu, K., Liu, J., Zheng, M., Zhuo, J., Zhang, S., Lin, D., Chen, K., et al. (2024) · Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2024.acl-long.515 · Cited by 32
ChatCoT: Tool-Augmented Chain-of-Thought Reasoning on Chat-based Large Language Models
Chen, Z., Zhou, K., Zhang, B., Gong, Z., Zhao, X., & Wen, J.-R. (2023) · Findings of the Association for Computational Linguistics: EMNLP 2023 · doi:10.18653/v1/2023.findings-emnlp.985 · Cited by 30
Middleware for LLMs: Tools Are Instrumental for Language Agents in Complex Environments
Gu, Y., Shu, Y., Yu, H., Liu, X., Dong, Y., Tang, J., Srinivasa, J., Latapie, H., & Su, Y. (2024) · Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2024.emnlp-main.436 · Cited by 28
StableToolBench: Towards Stable Large-Scale Benchmarking on Tool Learning of Large Language Models
Guo, Z., Cheng, S., Wang, H., Liang, S., Qin, Y., Li, P., Liu, Z., Sun, M., & Liu, Y. (2024) · arXiv · doi:10.48550/arxiv.2403.07714 · Cited by 22
Kong, Y., Ruan, J., Chen, Y., Zhang, B., Bao, T., Shiwei, S., Qing, D. G., Hu, X., Mao, H., Li, Z., et al. (2024) · Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: Industry Track · doi:10.18653/v1/2024.emnlp-industry.27 · Cited by 17
SciAgent: Tool-augmented Language Models for Scientific Reasoning
Ma, Y., Gou, Z., Hao, J., Xu, R., Wang, S., Pan, L., Yang, Y., Cao, Y., & Sun, A. (2024) · Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2024.emnlp-main.880 · Cited by 14
ToolNet: Connecting Large Language Models with Massive Tools via Tool Graph
Liu, X., Peng, Z., Yi, X., Xie, X., Xiang, L., Liu, Y., & Xu, D. (2024) · arXiv · doi:10.48550/arxiv.2403.00839 · Cited by 6
Tang, X., Liu, Y., Cai, Z., Shao, Y., Lu, J., Zhang, Y., Deng, Z., Hu, H., An, K., Huang, R., et al. (2023) · arXiv · doi:10.48550/arxiv.2311.09835 · Cited by 2 · cited 1×
“ML-BENCH [75] proposes a benchmark that evaluates LLMs’s ability to plan and execute machine learning workflows, such as building, training, and evaluating models, by composing code snippets and API calls.”
SciAgent: Tool-augmented Language Models for Scientific Reasoning
Ma, Y., Gou, Z., Hao, J., Xu, R., Wang, S., Pan, L., Yang, Y., Cao, Y., Sun, A., Awadalla, H., et al. (2024) · arXiv · doi:10.48550/arxiv.2402.11451 · Cited by 2
Mobile-Env: Building Qualified Evaluation Benchmarks for LLM-GUI Interaction
Zhang, D., Shen, Z., Xie, R., Zhang, S., Xie, T., Zhao, Z., Chen, S., Chen, L., Xu, H., Cao, R., et al. (2023) · arXiv · doi:10.48550/arxiv.2305.08144 · Cited by 1 · cited 3×
“20 Objective indicators Average-Steps [155] Result Check – Objective indicators Recall [126] Result Check Eq.”
Research works printed but not linked: 126
- Achiam J, Adler S, Agarwal S et al (2023) Gpt-4 technical report. ArXiv preprint arXiv:2303.08774 · arXiv
- Agarwal A, Chan A, Chandel S et al (2024) Copilot evaluation harness: evaluating llm-guided software programming. ArXiv preprint arXiv:2402.14261 · arXiv
- Ambrose SH (2001) Paleolithic technology and human evolution. Science 291(5509):1748–1753 · DOI · Paper page
- Anantha R, Bandyopadhyay B, Kashi A et al (2023) Protip: progressive tool retrieval improves planning. ArXiv preprint arXiv:2312.10332 · arXiv
- Barwise J (1977) An introduction to first-order logic. In: Studies in logic and the foundations of mathematics, vol 90. Elsevier, pp 5–46 · doi:10.1016/s0049-237x(08)71097-8
- Bhat V, Kaypak AU, Krishnamurthy P et al (2024) Grounding llms for robot task planning using closed-loop state feedback. CoRR · DOI
- Birr T, Pohl C, Younes A, et al (2024) Autogpt+ p: affordance-based task planning with large language models. ArXiv preprint arXiv:2402.10778 · DOI
- Cai T, Wang X, Ma T et al (2024) Large language models as tool makers. In: ICLR
- Chaslot GMJBC (2010) Monte-Carlo tree search. Doctoral Thesis
- Chen L, Shang S (2019) Region-based message exploration over spatio-temporal data streams. In: Proceedings of the AAAI conference on artificial intelligence, pp 873–880 · DOI
- Chen L, Shang S, Jensen CS, et al (2019) Effective and efficient reuse of past travel behavior for route recommendation. In: Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining, pp 488–498 · DOI
- Chen L, Lu K, Rajeswaran A et al (2021) Decision transformer: reinforcement learning via sequence modeling. In: Ranzato M, Yann N, Dauphin AB, Liang P et al (eds) Advances in neural information processing systems 34: annual conference on neural information processing systems 2021, NeurIPS 2021, December 6–14, 2021, virtual, pp 15084–15097. https://proceedings.neurips.cc/paper/2021/hash/7f489f642a0ddb10272b5c31057f0663-Abstract.html
- Chen W, Wang X, Wang WY (2023) A dataset for answering time-sensitive questions. In: Thirty-fifth conference on neural information processing systems datasets and benchmarks track (round 2)
- Chen Y, Liu Z, Li J et al (2022) Intent contrastive learning for sequential recommendation. IN: Proceedings of the ACM web conference, pp 2172–2182 · DOI · Paper page
- Chen Z, Du W, Zhang W et al (2023) T-eval: Evaluating the tool utilization capability step by step. ArXiv preprint arXiv:2312.14033 · arXiv
- Chen Z, Zhang D, Feng S et al (2024) Kgts: contrastive trajectory similarity learning over prompt knowledge graph embedding. In: Proceedings of the AAAI conference on artificial intelligence, pp 8311–8319 · DOI
- Cheong M, Abedin E, Ferreira M et al (2024) Investigating gender and racial biases in dall-e mini images. ACM J Responsib Comput 6:66
- Cobbe K, Kosaraju V, Bavarian M et al (2021) Training verifiers to solve math word problems. ArXiv preprint arXiv:2110.14168 · arXiv
- Dagan G, Keller F, Lascarides A (2023) Dynamic planning with a llm. ArXiv preprint arXiv:2308.06391 · arXiv
- Deng S, Xu W, Sun H et al (2024) Mobile-bench: an evaluation benchmark for llm-based mobile agents. In: Proceedings of the 62nd annual meeting of the association for computational linguistics (volume 1: long papers), pp 8813–8831 · DOI
- Ding T (2024) Mobileagent: enhancing mobile control via human-machine interaction and sop integration. ArXiv preprint arXiv:2401.04124 · arXiv
- Du C, Li Y, Qiu Z et al (2024) Stable diffusion is unstable. Adv Neural Inf Process Syst 36:66
- Du Y, Wei F, Zhang H (2024) Anytool: Self-reflective, hierarchical agents for large-scale api calls. In: International conference on machine learning (PMLR), pp 11812–11829
- Fan T, Kang Y, Ma G et al (2023) Fate-llm: a industrial grade federated learning framework for large language models. arXiv preprint arXiv:2310.10049 · arXiv
- Farn N, Shin R (2023) Tooltalk: evaluating tool-usage in a conversational setting. ArXiv preprint arXiv:2311.10775 · arXiv
- Fu D, Huang J, Lu S et al (2025) Preact: Prediction enhances agent’s planning ability. In: Proceedings of the 31st international conference on computational linguistics, pp 1–16
- Gao S, Shi Z, Zhu M et al (2024) Confucius: Iterative tool learning from introspection feedback by easy-to-difficult curriculum. In: Proceedings of the AAAI conference on artificial intelligence, pp 18030–18038 · DOI
- Gao S, Dwivedi-Yu J, Yu P et al (2025) Efficient tool use with chain-of-abstraction reasoning. In: COLING
- Gao Z, Du Y, Zhang X et al (2024b) Clova: a closed-loop visual assistant with tool usage and update. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 13258–13268 · DOI
- Ge Y, Hua W, Mei K et al (2024) Openagi: when llm meets domain experts. Adv Neural Inf Process Syst 36:66
- Gibson KR, Gibson KR, Ingold T (1993) Tools, language and cognition in human evolution. Cambridge University Press, Cambridge
- Gou Z, Shao Z, Gong Y et al (2023) Tora: a tool-integrated reasoning agent for mathematical problem solving. CoRR
- Grande V, Kiesler N, Francisco RMA (2024) Student perspectives on using a large language model (llm) for an assignment on professional ethics. In: Proceedings of the 2024 on innovation and technology in computer science education, vol 1, pp 478–484 · DOI
- Gui A, Li J, Dai Y et al (2024) Look before you leap: towards decision-aware and generalizable tool-usage for large language models. ArXiv preprint arXiv:2402.16696 · arXiv
- Han P, Yang P, Zhao P et al (2019) Gcn-mf: disease-gene association identification by graph convolutional networks and matrix factorization. In: Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining, pp 705–713 · DOI
- Hao S, Liu T, Wang Z et al (2023) Toolkengpt: augmenting frozen language models with massive tools via tool embeddings. Adv Neural Inf Process Syst 36:45870–45894
- He J, Chen J, He X et al (2016) Deep reinforcement learning with a natural language action space. In: Proceedings of the 54th annual meeting of the association for computational linguistics (volume 1: long papers). Association for Computational Linguistics, Berlin, Germany, pp 1621–1630. https://doi.org/10.18653/v1/P16-1153, https://aclanthology.org/P16-1153 · DOI · Paper page
- Hori K, Suzuki K, Ogata T (2024) Interactively robot action planning with uncertainty analysis and active questioning by large language model. In: 2024 IEEE/SICE international symposium on system integration (SII). IEEE, pp 85–91 · DOI
- Hsieh CY, Chen SA, Li CL et al (2023) Tool documentation enables zero-shot tool-usage with large language models. ArXiv preprint arXiv:2308.00675 · arXiv
- Hu EJ, Shen Y, Wallis P et al (2022) Lora: low-rank adaptation of large language models. In: The tenth international conference on learning representations, ICLR 2022, Virtual Event, April 25–29, 2022. OpenReview.net. https://openreview.net/forum?id=nZeVKeeFYf9
- Huang S, Zhong W, Lu J et al (2024) Planning, creation, usage: benchmarking llms for comprehensive tool utilization in real-world complex scenarios. Find Assoc Comput Linguist ACL 2024:4363–4400
- Huang X, Liu W, Chen X et al (2024) Understanding the planning of llm agents: a survey. CoRR
- Huang Y, Shi J, Li Y et al (2024) Metatool benchmark for large language models: deciding whether to use tools and which to use. In: ICLR
- Imrie F, Rauba P, van der Schaar M (2023) Redefining digital health interfaces with large language models. ArXiv preprint arXiv:2310.03560 · arXiv
- Jiang F, Peng Y, Dong L et al (2024) Large language model enhanced multi-agent systems for 6g communications. IEEE Wirel Commun 6:66
- Jin Q, Wang Z, Yang Y et al (2024) Agentmd: empowering language agents for risk prediction with large-scale clinical tool learning. ArXiv preprint arXiv:2402.13225 · DOI
- Kapania S, Wang R, Li TJJ et al (2024) “ i’m categorizing llm as a productivity tool”: examining ethics of llm use in hci research practices. arXiv preprint arXiv:2403.19876 · DOI
- Kumar A, Singh S, Murty SV et al (2024) The ethics of interaction: Mitigating security threats in llms. arXiv preprint arXiv:2401.12273 · arXiv
- Kumar SS, Jain D, Agarwal E et al (2024) Swissnyf: tool grounded llm agents for black box setting. ArXiv preprint arXiv:2402.10051 · arXiv
- Li J, Ye D, Shang S (2019) Adversarial transfer for named entity boundary detection with pointer networks. In: IJCAI, pp 5053–5059 · DOI
- Liu O, Fu D, Yogatama D et al (2024) Dellma: a framework for decision making under uncertainty with large language models. ArXiv preprint arXiv:2402.02392 · arXiv
- Liu S, Biswal A, Cheng A et al (2024) Optimizing llm queries in relational workloads. CoRR
- Liu S, Cheng H, Liu H et al (2024) Llava-plus: learning to use tools for creating multimodal agents. In: European conference on computer vision. Springer, Berlin, pp 126–142 · DOI
- Liu Y, Deng G, Xu Z et al (2023) Jailbreaking chatgpt via prompt engineering: an empirical study. arXiv preprint arXiv:2305.13860 · arXiv
- Liu Y, Peng X, Zhang Y et al (2024) Tool-planner: dynamic solution tree planning for large language model with tool clustering. arXiv preprint arXiv:2406.03807 · arXiv
- Liu Y, Yuan Y, Wang C et al (2024) From summary to action: Enhancing large language models for complex tasks with open world apis. ArXiv preprint arXiv:2402.18157 · arXiv
- Liu Z, Lai Z, Gao Z et al (2024) Controlllm: augment language models with tools by searching on graphs. In: European conference on computer vision. Springer, Berlin, pp 89–105 · DOI
- Long J (2023) Large language model guided tree-of-thought. ArXiv preprint arXiv:2305.08291 · arXiv
- Lu P, Peng B, Cheng H et al (2024) Chameleon: plug-and-play compositional reasoning with large language models. Adv Neural Inf Process Syst 36:66
- Lykov A, Konenkov M, Gbagbe KF et al (2024) Cognitiveos: large multimodal model based system to endow any type of robot with generative ai. ArXiv preprint arXiv:2401.16205 · DOI
- Lyu B, Cong X, Yu H et al (2023) Gitagent: Facilitating autonomous agent with github by tool extension. ArXiv preprint arXiv:2312.17294 · arXiv
- Ma X, Zhang Z, Zhao H (2024) Coco-agent: a comprehensive cognitive mllm agent for smartphone gui automation. Find Assoc Comput Linguist ACL 2024:9097–9110
- Mekala D, Weston J, Lanchantin J et al (2024) Toolverifier: generalization to new tools via self-verification. Find Assoc Comput Linguist EMNLP 2024:5026–5041
- Michael K, Abbas R, Roussos G et al (2020) Ethics in ai and autonomous system applications design. IEEE Trans Technol Soc 1(3):114–127 · DOI
- Moreira C, Calado P, Martins B (2015) Learning to rank academic experts in the dblp dataset. Expert Syst 32(4):477–493 · DOI
- Nakano R, Hilton J, Balaji S et al (2021) Webgpt: Browser-assisted question-answering with human feedback. ArXiv preprint arXiv:2112.09332 · arXiv
- Patil SG, Zhang T, Wang X et al (2024) Gorilla: large language model connected with massive apis. Adv Neural Inf Process Syst 37:126544–126565
- Pedro R, Castro D, Carreira P et al (2023) From prompt injections to sql injection attacks: How protected is your llm-integrated web application? ArXiv preprint arXiv:2308.01990 · arXiv
- Qian C, Xiong C, Liu Z et al (2024) Toolink: Linking toolkit creation and using through chain-of-solving on open-source model. In: Proceedings of the 2024 conference of the North American Chapter of the Association for Computational Linguistics: human language technologies (volume 1: long papers), pp 831–854 · DOI
- Qin Y, Liang S, Ye Y et al (2024) Toolllm: facilitating large language models to master 16000+ real-world apis. In: ICLR
- Rawles C, Li A, Rodriguez D et al (2023) Androidinthewild: a large-scale dataset for android device control. Adv Neural Inf Process Syst 36:59708–59728
- Roy D, Zhang X, Bhave R et al (2024) Exploring llm-based agents for root cause analysis. In: Companion proceedings of the 32nd ACM international conference on the foundations of software engineering, pp 208–219 · DOI · Paper page
- Ruan J, Chen Y, Zhang B et al (2023) Tptu: task planning and tool usage of large language model-based ai agents. In: NeurIPS 2023 foundation models for decision making workshop
- Ruan Y, Dong H, Wang A et al (2023) Identifying the risks of lm agents with an lm-emulated sandbox. In: NeurIPS 2023 foundation models for decision making workshop
- Schick T, Dwivedi-Yu J, Dessì R et al (2023) Toolformer: language models can teach themselves to use tools. Adv Neural Inf Process Syst 36:68539–68551
- Schick T, Dwivedi-Yu J, Dessì R et al (2024) Toolformer: language models can teach themselves to use tools. Adv Neural Inf Process Syst 36:66
- Shang S, Chen L, Zheng K et al (2018) Parallel trajectory-to-location join. IEEE Trans Knowl Data Eng 31(6):1194–1207 · DOI
- Shen W, Li C, Chen H et al (2024) Small llms are weak tool learners: a multi-llm agent. In: Proceedings of the 2024 conference on empirical methods in natural language processing, pp 16658–16680 · DOI
- Shen Y, Song K, Tan X et al (2024) Hugginggpt: solving ai tasks with chatgpt and its friends in hugging face. Adv Neural Inf Process Syst 36:66
- Shi Z, Gao S, Chen X et al (2024) Learning to use tools via cooperative and interactive agents. Find Assoc Comput Linguist EMNLP 2024:10642–10657
- Silver T, Hariprasad V, Shuttleworth RS et al (2022) Pddl planning with pretrained large language models. In: NeurIPS 2022 foundation models for decision making workshop
- Silver T, Dan S, Srinivas K et al (2024) Generalized planning in pddl domains with pretrained large language models. In: Proceedings of the AAAI conference on artificial intelligence, pp 20256–20264 · DOI
- Skrynnik A, Andreychuk A, Borzilov A et al (2024) Pogema: A benchmark platform for cooperative multi-agent navigation. arXiv preprint arXiv:2407.14931 · arXiv
- Song Y, Xiong W, Zhu D et al (2023) Restgpt: connecting large language models with real-world restful apis. ArXiv preprint arXiv:2306.06624 · arXiv
- Theuma A, Shareghi E (2024) Equipping language models with tool use capability for tabular data analysis in finance. In: Proceedings of the 18th conference of the european chapter of the Association for Computational Linguistics (volume 2: short papers), pp 90–103 · DOI
- Toubal IE, Avinash A, Alldrin NG et al (2024) Modeling collaborator: Enabling subjective vision classification with minimal human effort via llm tool-use. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 17553–17563 · DOI
- Toyama D, Hamel P, Gergely A et al (2021) Androidenv: a reinforcement learning platform for android. ArXiv preprint arXiv:2105.13231 · arXiv
- Wang B, Fang H, Eisner J et al (2024) Llms in the imaginarium: tool learning through simulated trial and error. In: Proceedings of the 62nd annual meeting of the association for computational linguistics (volume 1: long papers), pp 10583–10604 · DOI
- Wang C, Luo W, Chen Q et al (2024) Tool-lmm: a large multi-modal model for tool agent learning. ArXiv preprint arXiv:2401.10727 · arXiv
- Wang H, Wang H, Wang L et al (2023) Tpe: towards better compositional reasoning over conceptual tools with multi-persona collaboration. ArXiv preprint arXiv:2309.16090 · DOI
- Wang H, Feng S, Chen L et al (2024) Simulating individual infection risk over big trajectory data. In: International conference on database systems for advanced applications. Springer, Berlin, pp 136–151 · DOI
- Wang J, Shi E, Hu H et al (2024) Large language models for robotics: opportunities, challenges, and perspectives. J Autom Intell 6:66
- Wang J, Xu H, Ye J et al (2024) Mobile-agent: autonomous multi-modal mobile device agent with visual perception. In: ICLR 2024 workshop on large language model (LLM) agents
- Wang W, Shi J, Wang C et al (2024) Learning to ask: when llms meet unclear instruction. arXiv:2409.00557 · DOI
- Wang X, Wei J, Schuurmans D et al (2022) Self-consistency improves chain of thought reasoning in language models. ArXiv preprint arXiv:2203.11171 · arXiv
- Wang Y, Wu Z, Yao J et al (2025) Tdag: a multi-agent framework based on dynamic task decomposition and agent generation. Neural Netw 66:107200 · DOI
- Wei J, Wang X, Schuurmans D et al (2022) Chain-of-thought prompting elicits reasoning in large language models. Adv Neural Inf Process Syst 35:24824–24837
- Wei Y, Su Y, Ma H et al (2023) Menatqa: a new dataset for testing the temporal comprehension and reasoning abilities of large language models. In: Findings of the Association for Computational Linguistics: EMNLP 2023, pp 1434–1447 · DOI
- Wu Q, Xu W, Liu W et al (2024) Mobilevlm: a vision-language model for better intra-and inter-ui understanding. arXiv preprint arXiv:2409.14818 · DOI
- Xie J, Chen Z, Zhang R et al (2024) Large multimodal agents: a survey. ArXiv preprint arXiv:2402.15116 · arXiv
- Xu Q, Hong F, Li B et al (2023) On the tool manipulation capability of open-sourced large language models. In: NeurIPS 2023 foundation models for decision making workshop
- Yang C, Chen L, Shang S et al (2019) Toward efficient navigation of massive-scale geo-textual streams. In: IJCAI, pp 4838–4845 · doi:10.24963/ijcai.2019/672
- Yang R, Song L, Li Y et al (2024) Gpt4tools: teaching large language model to use tools via self-instruction. Adv Neural Inf Process Syst 36:66
- Yang Z, Liu J, Han Y et al (2023) Appagent: Multimodal agents as smartphone users. ArXiv preprint arXiv:2312.13771 · arXiv
- Yao S, Yu D, Zhao J et al (2023) Tree of thoughts: deliberate problem solving with large language models. Adv Neural Inf Process Syst 36:11809–11822
- Yao S, Zhao J, Yu D et al (2023) React: synergizing reasoning and acting in language models. In: International conference on learning representations (ICLR)
- Ye J, Li S, Li G et al (2024) Toolsword: Unveiling safety issues of large language models in tool learning across three stages. In: Proceedings of the 62nd annual meeting of the association for computational linguistics (volume 1: long papers), pp 2181–2211 · DOI
- Ye J, Wu Y, Gao S et al (2024) Rotbench: a multi-level benchmark for evaluating the robustness of large language models in tool learning. In: Proceedings of the 2024 conference on empirical methods in natural language processing, pp 313–333 · DOI
- Ye J, Li G, Gao S et al (2025) Tooleyes: fine-grained evaluation for tool learning capabilities of large language models in real-world scenarios. In: Proceedings of the 31st international conference on computational linguistics, pp 156–187
- Ye X, Chen Q, Dillig I et al (2024) Satlm: satisfiability-aided language models using declarative prompting. Adv Neural Inf Process Syst 36:66
- Yuan S, Song K, Chen J et al (2024) Easytool: enhancing llm-based agents with concise tool instruction. In: ICLR 2024 workshop on large language model (LLM) agents
- Zhang D, Chen L, Yu K (2023) Mobile-env: a universal platform for training and evaluation of mobile interaction. CoRR
- Zhang J (2023) Graph-toolformer: to empower llms with graph reasoning ability via prompt augmented by chatgpt. ArXiv preprint arXiv:2304.11116 · arXiv
- Zhang Z, Zhang A (2024) You only look at screens: Multimodal chain-of-action agents. In: Findings of the Association for Computational Linguistics ACL 2024, pp 3132–3149 · DOI · Paper page
- Zhao H, Liu Z, Wu Z et al (2024) Revolutionizing finance with llms: an overview of applications and insights. arXiv preprint arXiv:2401.11641 · arXiv
- Zhao L, Yang Y, Zhang K et al (2024) Diffagent: fast and accurate text-to-image api selection with large language model. In: 2024 IEEE/CVF conference on computer vision and pattern recognition (CVPR). IEEE Computer Society, pp 6390–6399 · DOI
- Zhao Q, Zhou X, Wu J et al (2024) Biotreasury: a community-based repository enabling indexing and rating of bioinformatics tools. Sci China Life Sci 67(2):221–229 · DOI
- Zhao Y, Xia J, Liu G et al (2019) Preference-aware task assignment in spatial crowdsourcing. In: Proceedings of the AAAI conference on artificial intelligence, pp 2629–2636 · doi:10.1609/aaai.v33i01.33012629
- Zheng B, Gou B, Kil J et al (2024) Gpt-4v (ision) is a generalist web agent, if grounded. In: International conference on machine learning, PMLR, pp 61349–61385
- Zheng Y, Li P, Liu W et al (2024) Toolrerank: adaptive and hierarchy-aware reranking for tool retrieval. In: Proceedings of the 2024 joint international conference on computational linguistics, language resources and evaluation (LREC-COLING 2024), pp 16263–16273
- Zheng Y, Li P, Yan M et al (2024) Budget-constrained tool learning with planning. In: Findings of the Association for Computational Linguistics ACL 2024, pp 9039–9052 · DOI
- Zhou X, Zhao X, Li G (2024) Llm-enhanced data management. ArXiv preprint arXiv:2402.02643 · arXiv
- Zhu JY, Cano CG, Bermudez DV et al (2024) Incoro: in-context learning for robotics control with feedback loops. ArXiv preprint arXiv:2402.05188 · arXiv
- Zhuang Y, Yu Y, Wang K et al (2023) Toolqa: a dataset for llm question answering with external tools. In: Proceedings of the 37th international conference on neural information processing systems, pp 50117–50143
- Zhuang Y, Chen X, Yu T et al (2024) Toolchain*: efficient action space navigation in large language models with a* search. In: The twelfth international conference on learning representations
- Zou A, Wang Z, Carlini N et al (2023) Universal and transferable adversarial attacks on aligned language models. arXiv preprint arXiv:2307.15043 · arXiv
Cited by
From Static Templates to Dynamic Runtime Graphs: A Survey of Workflow Optimization for LLM Agents
Yue, L., Bhandari, K. R., Ko, C.-Y., Patel, D., Lin, S., Zhou, N., Gao, J., Chen, P.-Y., & Pan, S. (2026) · arXiv · doi:10.48550/arxiv.2603.22386 · Cited by 1
“Tool-learning surveys focus on retrieving, selecting, and invoking tools (Xu et al., 2025b).”
Caring Without Feeling: Affective Dynamics as the Control Layer of Human-AI Agent Collaboration
Xu, J., Wu, X., Zhang, Z., Xu, Y., Yang, Y., Zhu, J., Xiao, L., Wu, W., & He, L. (2026) · arXiv · doi:10.48550/arxiv.2606.18259
“Failure detection must occur in real time as early mistakes compound [73, 74].”
MemMorph: Tool Hijacking in LLM Agents via Memory Poisoning
Zhang, X., Zheng, Y., Xu, Z., Zhou, K., Shen, B., Ou, H., Zhang, T., & Lam, K.-Y. (2026) · arXiv · doi:10.48550/arxiv.2605.26154
“In such tool-augmented setting, agent’s performance highly depends on making correct action-routing decisions, including planing multi-step behaviors, de-composing tasks, and selecting an appropriate tool at each step (Xu et al., 2025).”
Metadata
- Authors
- Xu, W., Huang, C., Gao, S., & Shang, S.
- Venue
- Data Science and Engineering
- Published
- 2025-06-26
- Volume
- 10
- Issue
- 4
- Pages
- 533-563
- ISSN
- 2364-1185
- 2364-1541
- Cited by
- 74
- References
- 52
- Language
- en
- Licence
- creativecommons.org/licenses/by/4.0 (tdm)
- creativecommons.org/licenses/by/4.0 (vor)
- Funders
- Natural Science Foundation of China (62406061)
- National Natural Science Foundation of China (62432002)
- Natural Science Foundation of Shandong Province (ZR2023QF159)
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.