MetaTool Benchmark for Large Language Models: Deciding Whether to Use Tools and Which to Use
Open accessAbstract
Large language models (LLMs) have garnered significant attention due to their impressive natural language processing (NLP) capabilities. Recently, many studies have focused on the tool utilization ability of LLMs. They primarily investigated how LLMs effectively collaborate with given specific tools. However, in scenarios where LLMs serve as intelligent agents, as seen in applications like AutoGPT and MetaGPT, LLMs are expected to engage in intricate decision-making processes that involve deciding whether to employ a tool and selecting the most suitable tool(s) from a collection of available tools to fulfill user requests. Therefore, in this paper, we introduce MetaTool, a benchmark designed to evaluate whether LLMs have tool usage awareness and can correctly choose tools. Specifically, we create a dataset called ToolE within the benchmark. This dataset contains various types of user queries in the form of prompts that trigger LLMs to use tools, including both single-tool and multi-tool scenarios. Subsequently, we set the tasks for both tool usage awareness and tool selection. We define four subtasks from different perspectives in tool selection, including tool selection with similar choices, tool selection in specific scenarios, tool selection with possible reliability issues, and multi-tool selection. We conduct experiments involving eight popular LLMs and find that the majority of them still struggle to effectively select tools, highlighting the existing gaps between LLMs and genuine intelligent agents. However, through the error analysis, we found there is still significant room for improvement. Finally, we conclude with insights for tool developers -- we strongly recommend that tool developers choose an appropriate rewrite model for generating new descriptions based on the downstream LLM the tool will apply to. Our code is in https://github.com/HowieHwong/MetaTool.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Datasets such as FlowBench, ToolBench, and API-Bank focus on tool use and function-calling across large API repositories.
“Others, such as FlowBench [96], ToolBench [38], and API-Bank [47], focus on tool use and function-calling across large API repositories.”
From Evaluation and Benchmarking of LLM Agents: A Survey (2025) · cites this paper as [38]
In the “Dimension APIBank GPT4Tool” comparison table (page unavailable), the paper lists ToolBench and MetaTool as separate benchmarks and labels MetaTool “Ours.” It therefore contradicts the identification implied by citing this paper as ToolBench; the passage also does not establish the claim about function-calling across large API repositories. (LLM, conf=0.85)
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredBFCL, ToolBench, and MetaTool test tool use/function calling in multiple programming languages and propose methods for evaluating the accuracy of function calls.
“Projects like the Berkeley Function Calling Leaderboard (BFCL) [23], ToolBench [22] and MetaTool [11] test tool use/function calling in multiple programming languages and propose various methods for evaluating the accuracy of function calls.”
From $τ$-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains (2024) · cites this paper as [11]
The Conclusion identifies MetaTool as a benchmark for tool-use awareness and tool selection, and the Metrics section proposes Correct Selection Rate to measure whether an LLM chooses the right tools (page numbers unavailable). These passages support testing tool use and evaluating selection accuracy, but not testing across multiple programming languages or evaluating the accuracy of executed function calls. (LLM, conf=0.87)
Experimental — accuracy not yet measuredPrior tool-use benchmarks only contain single-step user interaction, where the human provides an initial instruction containing all required information.
“However, all these works only contain a single-step user interaction, where the human interacting with the agent provides an initial instruction containing all the required information.”
From $τ$-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains (2024) · cites this paper as [11]
The dataset comparison section says earlier datasets lacked diverse user inputs, and §2.2 describes MetaTool’s own data as user requests paired with tools. Neither passage establishes that all prior tool-use benchmarks had only single-step user interaction or that their initial instructions contained all required information. Searches for single-step interaction, multi-turn dialogue, and follow-up information found no passage addressing that claim. (LLM, conf=0.72)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
The publisher hasn’t deposited this paper’s reference list.
Research works printed but not linked: 7
- Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. Survey of hallucination in natural language generation. ACM Computing Surveys, 55(12):1-38, 2023.
- Peter J Rousseeuw. Silhouettes: a graphical aid to the interpretation and validation of cluster analysis. Journal of computational and applied mathematics, 20:53-65, 1987.
- Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(11), 2008.
- Haochun Wang, Chi Liu, Nuwa Xi, Zewen Qiang, Sendong Zhao, Bing Qin, and Ting Liu. Huatuo: Tuning llama model with chinese medical knowledge. arXiv preprint arXiv:2304.06975, 2023c. · arXiv
- Shijie Wu, Ozan Irsoy, Steven Lu, Vadim Dabravolski, Mark Dredze, Sebastian Gehrmann, Prabhan- jan Kambadur, David Rosenberg, and Gideon Mann. Bloomberggpt: A large language model for finance. arXiv preprint arXiv:2303.17564, 2023b. · arXiv
- Daniel Xiao. Ai-based literature review tools, 2023. https://tamu.libguides.com/c.php?g=1289555.
- Hongyang Yang, Xiao-Yang Liu, and Christina Dan Wang. Fingpt: Open-source financial large language models. arXiv preprint arXiv:2306.06031, 2023b. · arXiv
Cited by
Sorted by citations
Evaluation and Benchmarking of LLM Agents: A Survey
Mohammadi, M., Li, Y., Lo, J., & Yip, W. (2025) · Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.2 · doi:10.1145/3711896.3736570 · Cited by 74 · see citing works
Self-citationJustice or Prejudice? Quantifying Biases in LLM-as-a-Judge
Ye, J., Wang, Y., Huang, Y., Chen, D., Zhang, Q., Moniz, N., Gao, T., Geyer, W., Huang, C., Chen, P.-Y., et al. (2024) · arXiv · doi:10.48550/arxiv.2410.02736 · Cited by 11 · see citing works
Self-citationGenerative AI Agents for Knowledge Work Augmentation in Finance
Ganesh, S., Ardon, L., Borrajo, D., Garg, D., Madhushani, U., Narayanan, A. L., Canonaco, G., & Veloso, M. (2024) · Annual Review of Control Robotics and Autonomous Systems · doi:10.1146/annurev-control-022823-033813 · Cited by 10 · see citing works
DAWN: Designing Distributed Agents in a Worldwide Network
Aminiranjbar, Z., Tang, J., Wang, Q., Pant, S., & Viswanathan, M. (2024) · arXiv · doi:10.48550/arxiv.2410.22339 · Cited by 8 · see citing works
DAWN: Designing Distributed Agents in a Worldwide Network
Aminiranjbar, Z., Tang, J., Wang, Q., Pant, S., & Viswanathan, M. (2025) · IEEE Access · doi:10.1109/access.2025.3588425 · Cited by 8 · see citing works
$τ$-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains
Yao, S., Shinn, N., Razavi, P., & Narasimhan, K. (2024) · arXiv · doi:10.48550/arxiv.2406.12045 · Cited by 9 · see citing works
“Projects like the Berkeley Function Calling Leaderboard (BFCL) [23], ToolBench [22] and MetaTool [11] test tool use/function calling in multiple programming languages and propose various methods for evaluating the accuracy of function calls.”
MethodFang, J., Peng, Y., Zhang, X., Wang, Y., Yi, X., Zhang, G., Xu, Y., Wu, B., Liu, S., Li, Z., et al. (2025) · arXiv · doi:10.48550/arxiv.2508.07407 · Cited by 6 · see citing works
When2Call: When (not) to Call Tools
Ross, H., Mahabaleshwarkar, A. S., & Suhara, Y. (2025) · Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) · doi:10.18653/v1/2025.naacl-long.174 · Cited by 4
$τ^2$-Bench: Evaluating Conversational Agents in a Dual-Control Environment
Barres, V., Dong, H., Ray, S., Si, X., & Narasimhan, K. (2025) · arXiv · doi:10.48550/arxiv.2506.07982 · Cited by 1 · see citing works
“Following a long line of research into language agents [25, 27, 10, 13, 21], LLM tool use [24, 20, 9], and task-oriented dialog [5, 4, 1, 22, 6, 7, 8], τ -bench [26] is a recently introduced benchmark to measure the reliability of language agents in multi-turn task-oriented conversations such as…”
SOLAR: Illuminating LLM performance in API discovery and service ranking for edge AI and IoT
Al‐Masri, E., & Subramanian, I. N. (2025) · Internet of Things · doi:10.1016/j.iot.2025.101630 · Cited by 3 · see citing works
Preference Leakage: A Contamination Problem in LLM-as-a-judge
Li, D., Sun, R., Huang, Y., Zhong, M., Jiang, B., Han, J., Zhang, X., Wang, W., & Liu, H. (2025) · arXiv · doi:10.48550/arxiv.2502.01534 · Cited by 2 · see citing works
“Representatively, LLM-based AI agents’ focus transfer from traditional natural language processing tasks [70, 73] to real-world [46, 20], open-ended response generation [65], which greatly limits the applicability of traditional n-gram matching methods (e.g., BLEU [50] and ROUGE [41]) [43, 53] or…”
Self-citationWTU-EVAL: A Whether-or-Not Tool Usage Evaluation Benchmark for Large Language Models
Liu, J., Ning, K., Su, Y., Han, W., Xu, J., & Zhang, Y. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10889153 · Cited by 8 · see citing works
Advanced Statistical Techniques in Healthcare: Innovations in Data Analysis and Decision Making
Saleh, M. A. H. H., Awad, O. R. D., bizun, H. K. M., & Oufi, M. R. H. (2025) · Bright Sky Publications eBooks · doi:10.62906/bs.book.293 · Cited by 1 · see citing works
Re-Invoke: Tool Invocation Rewriting for Zero-Shot Tool Retrieval
Chen, Y., Yoon, J., Sachan, D. S., Wang, Q., Cohen-Addad, V., Bateni, M., Lee, C.-Y., & Pfister, T. (2024) · arXiv · doi:10.48550/arxiv.2408.01875
“The round-trip consistency recall metric is lower on ToolBench I1 and I2 datasets, mainly caused by the larger toolset size compared to ToolBench I3 and ToolE datasets.”
InfluentialMemMorph: Tool Hijacking in LLM Agents via Memory Poisoning
Zhang, X., Zheng, Y., Xu, Z., Zhou, K., Shen, B., Ou, H., Zhang, T., & Lam, K.-Y. (2026) · arXiv · doi:10.48550/arxiv.2605.26154
Enrich-Retrieve-Rank: Scaling Capability Discovery Beyond In-Context Routing
Sorathiya, N., Zhang, D., & Akhbari, B. (2026) · arXiv · doi:10.48550/arxiv.2608.22695
AJ-Bench: Benchmarking Agent-as-a-Judge for Environment-Aware Evaluation
Shi, W., Wang, Y., Zhao, Y., Chen, Y., Feng, F., Hao, X., Su, X., Gu, Q., Su, H., Cai, X., et al. (2026) · arXiv · doi:10.48550/arxiv.2604.18240
NoRMA: A Multi-agent Communication-Centric Dataset for Enhanced Customs Nomenclature Classification
Bouchtib, H., Karboub, K., Tabaa, M., & Hamlich, M. (2026) · Communications in computer and information science · doi:10.1007/978-3-032-15632-7_18
The Fifth Industrial Revolution: A New Cognitive Era & Unprecedented Human-AI Collaboration
Woodring, D. (2026) · Springer eBooks · doi:10.1007/978-3-032-05205-6_1 · Cited by 1 · see citing works
Prompt Injection Attack to Tool Selection in LLM Agents
Shi, J., Yuan, Z., Tie, G., Zhou, P., Gong, N. Z., & Sun, L. (2025) · arXiv · doi:10.48550/arxiv.2504.19793 · Cited by 2 · see citing works
Self-citation
- Authors
- Huang, Y., Shi, J., Li, Y., Fan, C., Wu, S., Zhang, Q., Liu, Y., Zhou, P., Wan, Y., Gong, N. Z., & Sun, L.
- Affiliations
- Lehigh University (US)
- Huazhong University of Science and Technology (CN)
- University of Cambridge (GB)
- Duke University (US)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2023
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2310.03128
- OpenAlex
- W4387427530
- Cited by
- 7
- References
- 0
- Citations by year
- 2026: 2 · 2025: 4 · 2024: 1
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- arXiv (Cornell University) (submittedVersion, cc-by-nc-sa)
- arXiv (Cornell University)
- Funders
- National Science Foundation (2246067)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Keywords
- large language models, tool selection, LLM agents, tool reliability, prompt engineering, intelligent agents
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
