API-Bank: A Comprehensive Benchmark for Tool-Augmented LLMs
Open accessAbstract
Recent research has demonstrated that Large Language Models (LLMs) can enhance their capabilities by utilizing external tools. However, three pivotal questions remain unanswered: (1) How effective are current LLMs in utilizing tools? (2) How can we enhance LLMs' ability to utilize tools? (3) What obstacles need to be overcome to leverage tools? To address these questions, we introduce API-Bank, a groundbreaking benchmark, specifically designed for tool-augmented LLMs. For the first question, we develop a runnable evaluation system consisting of 73 API tools. We annotate 314 tool-use dialogues with 753 API calls to assess the existing LLMs' capabilities in planning, retrieving, and calling APIs. For the second question, we construct a comprehensive training set containing 1,888 tool-use dialogues from 2,138 APIs spanning 1,000 distinct domains. Using this dataset, we train Lynx, a tool-augmented LLM initialized from Alpaca. Experimental results demonstrate that GPT-3.5 exhibits improved tool utilization compared to GPT-3, while GPT-4 excels in planning. However, there is still significant potential for further improvement. Moreover, Lynx surpasses Alpaca's tool utilization performance by more than 26 pts and approaches the effectiveness of GPT-3.5. Through error analysis, we highlight the key challenges for future research in this field to answer the third question.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
LLM agents can directly build on the functions of the underlying model, allowing many evaluation techniques originally developed for LLMs to use tools.
“As a result, LLM agents can directly build on the functions of the underlying model, allowing many of the evaluation techniques originally developed for LLMs to use tools [47].”
From Evaluation and Benchmarking of LLM Agents: A Survey (2025) · cites this paper as [47]
Section 3.3 (page unavailable) evaluates tool-augmented LLM responses with ROUGE-L, supporting the use of an existing language-model evaluation metric in this setting. The paper does not establish that many LLM evaluation techniques transfer to agents, or make the broader claim about agents building directly on the underlying model's functions. (LLM, conf=0.78)
Experimental — accuracy not yet measuredDatasets such as FlowBench, ToolBench, and API-Bank focus on tool use and function-calling across large API repositories.
“Others, such as FlowBench [96], ToolBench [38], and API-Bank [47], focus on tool use and function-calling across large API repositories.”
From Evaluation and Benchmarking of LLM Agents: A Survey (2025) · cites this paper as [47]
API-Bank comprises a comprehensive Tool-Augmented LLM workflow, encompassing 53 commonly used API tools and 264 annotated dialogues, encompassing a total of 568 API calls.
“It comprises a comprehensive Tool-Augmented LLM workflow, encompassing 53 commonly used API tools and 264 annotated dialogues, encompassing a total of 568 API calls.”
From A Survey on Evaluation of Large Language Models (2024) · cites this paper as [109]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · see citing works
LLaMA: Open and Efficient Foundation Language Models
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. (2023) · arXiv · doi:10.48550/arxiv.2302.13971 · Cited by 4,602 · see citing works
Sparks of Artificial General Intelligence: Early experiments with GPT-4
Bubeck, S., Chandrasekaran, V., Eldan, R., Gehrke, J., Horvitz, E., Kamar, E., Lee, P., Lee, Y. T., Li, Y., Lundberg, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.12712 · Cited by 1,526 · see citing works
Self-citationEvaluating Large Language Models Trained on Code
Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. D. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. (2021) · arXiv · doi:10.48550/arxiv.2107.03374 · Cited by 1,436 · see citing works
Toolformer: Language Models Can Teach Themselves to Use Tools
Schick, T., Dwivedi-Yu, J., Dessì, R., Raileanu, R., Lomeli, M., Zettlemoyer, L., Cancedda, N., & Scialom, T. (2023) · arXiv · doi:10.48550/arxiv.2302.04761 · Cited by 1,150 · see citing works
Toolformer: language models can teach themselves to use tools
Schick, T., Dwivedi-Yu, J., Dessi, R., Raileanu, R., Lomeli, M., Hambro, E., Zettlemoyer, L., Cancedda, N., & Scialom, T. (2023) · Advances in Neural Information Processing Systems 36 · doi:10.52202/075280-2997 · Cited by 1,150 · see citing works
GLM: General Language Model Pretraining with Autoregressive Blank Infilling
Du, Z., Qian, Y., Liu, X., Ding, M., Qiu, J., Yang, Z., & Tang, J. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.26 · Cited by 1,028 · see citing works
Paleolithic Technology and Human Evolution
Ambrose, S. H. (2001) · Science · doi:10.1126/science.1059487 · Cited by 773 · see citing works
ReAct: Synergizing Reasoning and Acting in Language Models
Yao, S., Zhao, J., Yu, D., Du, N., Shafran, I., Narasimhan, K., & Cao, Y. (2022) · arXiv · doi:10.48550/arxiv.2210.03629 · Cited by 608 · see citing works
ToolQA: A Dataset for LLM Question Answering with External ToolsPreprint
Zhuang, Y.-C., Yu, Y., Wang, K., et al. (2023) · Neural Information Processing Systems · doi:10.48550/arxiv.2306.13304 · Cited by 421 · see citing works
TaskMatrix.AI: Completing Tasks by Connecting Foundation Models with Millions of APIsPreprint
Liang, Y., Wu, C., Song, T., et al. (2023) · Intelligent Computing · doi:10.48550/arxiv.2303.16434 · Cited by 282 · see citing works
WebGPT: Browser-assisted question-answering with human feedback
Nakano, R., Hilton, J., Balaji, S., Wu, J., Ouyang, L., Kim, C., Hesse, C., Jain, S., Kosaraju, V., Saunders, W., et al. (2021) · arXiv · doi:10.48550/arxiv.2112.09332 · Cited by 272 · see citing works
ART: Automatic multi-step reasoning and tool-use for large language modelsPreprint
Paranjape, B., Lundberg, S. M., Singh, S., Hajishirzi, H., Zettlemoyer, L., & Ribeiro, M. T. (2023) · arXiv.org · doi:10.48550/arxiv.2303.09014 · Cited by 239 · see citing works
Gorilla: Large Language Model Connected with Massive APIs
Patil, S. G., Zhang, T., Wang, X., & Gonzalez, J. E. (2023) · arXiv · doi:10.48550/arxiv.2305.15334 · Cited by 226 · see citing works
Gorilla: Large Language Model Connected with Massive APIs
Patil, S., Zhang, T., Wang, X., & Gonzalez, J. (2024) · Advances in Neural Information Processing Systems 37 · doi:10.52202/079017-4020 · Cited by 226 · see citing works
Atlas: Few-shot Learning with Retrieval Augmented Language Models
Izacard, G., Lewis, P., Lomeli, M., Hosseini, L., Petroni, F., Schick, T., Dwivedi-Yu, J., Joulin, A., Riedel, S., & Grave, E. (2022) · arXiv · doi:10.48550/arxiv.2208.03299 · Cited by 213 · see citing works
Socratic Models: Composing Zero-Shot Multimodal Reasoning with Language
Zeng, A., Attarian, M., Ichter, B., Choromanski, K., Wong, A., Welker, S., Tombari, F., Purohit, A., Ryoo, M., Sindhwani, V., et al. (2022) · arXiv · doi:10.48550/arxiv.2204.00598 · Cited by 205 · see citing works
Tool Learning with Foundation Models
Qin, Y., Hu, S., Lin, Y., Chen, W., Ding, N., Cui, G., Zeng, Z., Huang, Y., Xiao, C., Han, C., et al. (2023) · arXiv · doi:10.48550/arxiv.2304.08354 · Cited by 184 · see citing works
(2024) · doi:10.1145/3704435 · Cited by 184 · see citing works
Augmented Language Models: a Survey
Mialon, G., Dessì, R., Lomeli, M., Nalmpantis, C., Pasunuru, R., Raileanu, R., Rozière, B., Schick, T., Dwivedi-Yu, J., Celikyilmaz, A., et al. (2023) · arXiv · doi:10.48550/arxiv.2302.07842 · Cited by 160 · see citing works
Self-Instruct: Aligning Language Models with Self-Generated Instructions
Wang, Y., Kordi, Y., Mishra, S., Liu, A., Smith, N. A., Khashabi, D., & Hajishirzi, H. (2022) · arXiv · doi:10.48550/arxiv.2212.10560 · Cited by 118 · see citing works
TaskMatrix.AI: Completing Tasks by Connecting Foundation Models with Millions of APIs
Liang, Y., Wu, C., Song, T., Wu, W., Xia, Y., Liu, Y., Ou, Y., Lu, S., Ji, L., Mao, S., et al. (2024) · Intelligent Computing · doi:10.34133/icomputing.0063 · Cited by 108 · see citing works
Making Language Models Better Tool Learners with Execution FeedbackPreprint
Qiao, S.-F., Gui, H., Lv, C.-F., Jia, Q.-H., Chen, H.-J., & Zhang, N. (2023) · North American Chapter of the Association for Computational Linguistics · doi:10.48550/arxiv.2305.13068 · Cited by 93 · see citing works
Preference Ranking Optimization for Human Alignment
Song, F., Yu, B., Li, M., Yu, H., Huang, F., Li, Y., & Wang, H. (2023) · arXiv · doi:10.48550/arxiv.2306.17492 · Cited by 77 · see citing works
Self-citationPreference Ranking Optimization for Human Alignment
Song, F., Yu, B., Li, M., Yu, H., Huang, F., Li, Y., & Wang, H. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i17.29865 · Cited by 77 · see citing works
Self-citationToolQA: A Dataset for LLM Question Answering with External Tools
Zhuang, Y., Yu, Y., Wang, K., Sun, H., & Zhang, C. (2023) · neural information processing systems · doi:10.52202/075280-2180 · Cited by 73 · see citing works
Qin, Y., Liang, S., Ye, Y., Zhu, K., Yan, L., Lu, Y., Lin, Y., Cong, X., Tang, X., Qian, B., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2307.16789 · Cited by 71 · see citing works
ToolkenGPT: Augmenting Frozen Language Models with Massive Tools via Tool Embeddings
Hao, S., Liu, T., Wang, Z., & Hu, Z. (2023) · arXiv · doi:10.48550/arxiv.2305.11554 · Cited by 35 · see citing works
ToolkenGPT: Augmenting Frozen Language Models with Massive Tools via Tool Embeddings
Hao, S., Liu, T., Wang, Z., & Hu, Z. (2023) · neural information processing systems · doi:10.52202/075280-1988 · Cited by 35 · see citing works
Large language models as tool makers
Cai, T., Wang, X., Ma, T., Chen, X., & Zhou, D. (2023) · arXiv · doi:10.48550/arxiv.2305.17126 · Cited by 28 · cited 1× · see citing works
“LATM(Cai et al., 2023) and CREATOR (Qian et al., 2023) leverage GPT-4 to make API tools.”
BackgroundCREATOR: Tool Creation for Disentangling Abstract and Concrete Reasoning of Large Language Models
Qian, C., Han, C., Fung, Y., Qin, Y., Liu, Z., & Ji, H. (2023) · Findings of the Association for Computational Linguistics: EMNLP 2023 · doi:10.18653/v1/2023.findings-emnlp.462 · Cited by 24 · see citing works
ToolAlpaca: Generalized Tool Learning for Language Models with 3000 Simulated Cases
Tang, Q., Deng, Z., Lin, H., Han, X., Liang, Q., Cao, B., & Sun, L. (2023) · arXiv · doi:10.48550/arxiv.2306.05301 · Cited by 16 · see citing works
On the Tool Manipulation Capability of Open-source Large Language Models
Xu, Q., Hong, F., Li, B., Hu, C., Chen, Z., & Zhang, J. (2023) · arXiv · doi:10.48550/arxiv.2305.16504 · Cited by 11 · see citing works
A Preliminary Study of the Intrinsic Relationship between Complexity and AlignmentPreprint
Zhao, Y., Yu, B., Hui, B., Yu, H., Huang, F., Li, Y., & Zhang, N. L. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2308.05696 · Cited by 4 · see citing works
Self-citation
Flagged first, then by citations
Research works printed but not linked: 1
- Cheng Qian, Chi Han, Yi R Fung, Yujia Qin, Zhiyuan Liu, and Heng Ji. 2023. Creator: Disentan- gling abstract and concrete reasonings of large lan- guage models through tool creation. arXiv preprint arXiv:2305.14318. · arXiv
Cited by
Sorted by citations
A Survey on Evaluation of Large Language Models
Chang, Y., Wang, X., Wang, J., Wu, Y., Yang, L., Zhu, K., Chen, H., Yi, X., Wang, C., Wang, Y., et al. (2024) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3641289 · Cited by 2,842 · see citing works
A survey on large language model based autonomous agents
Wang, L., Ma, C., Feng, X., Zhang, Z., Yang, H., Zhang, J., Chen, Z., Tang, J., Chen, X., Lin, Y., et al. (2024) · Frontiers of Computer Science · doi:10.1007/s11704-024-40231-1 · Cited by 1,941 · see citing works
Mind2Web: Towards a Generalist Agent for the Web
Deng, X., Gu, Y., Zheng, B., Chen, S., Stevens, S., Wang, B., Sun, H., & Su, Y. (2023) · arXiv · doi:10.48550/arxiv.2306.06070 · Cited by 134 · see citing works
Datasets for large language models: a comprehensive survey
Liu, Y., Cao, J., Liu, C., Ding, K., & Jin, L. (2025) · Artificial Intelligence Review · doi:10.1007/s10462-025-11403-7 · Cited by 159 · see citing works
Tool learning with large language models: a survey
Qu, C., Dai, S., Wei, X., Cai, H., Wang, S., Yin, D., Xu, J., & Wen, J.-R. (2025) · Frontiers of Computer Science · doi:10.1007/s11704-024-40678-2 · Cited by 148 · see citing works
Inadequacies of Large Language Model Benchmarks in the Era of Generative Artificial Intelligence
McIntosh, T. R., Susnjak, T., Arachchilage, N., Liu, T., Xu, D., Watters, P., & Halgamuge, M. N. (2026) · IEEE Transactions on Artificial Intelligence · doi:10.1109/tai.2025.3569516 · Cited by 104 · see citing works
RCAgent: Cloud Root Cause Analysis by Autonomous Agents with Tool-Augmented Large Language Models
Wang, Z., Liu, Z., Zhang, Y., Zhong, A., Wang, J., Yin, F., Fan, L., Wu, L., & Wen, Q. (2024) · Proceedings of the 33rd ACM International Conference on Information and Knowledge Management · doi:10.1145/3627673.3680016 · Cited by 99 · see citing works
Preference Ranking Optimization for Human Alignment
Song, F., Yu, B., Li, M., Yu, H., Huang, F., Li, Y., & Wang, H. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i17.29865 · Cited by 77 · see citing works
Self-citationQin, Y., Liang, S., Ye, Y., Zhu, K., Yan, L., Lu, Y., Lin, Y., Cong, X., Tang, X., Qian, B., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2307.16789 · Cited by 71 · see citing works
AIGC-empowered smart manufacturing: Prospects and challenges
Leng, J., Zheng, K., Li, R. C., Chen, C., Wang, B., Liu, Q., Chen, X., & Shen, W. (2025) · Robotics and Computer-Integrated Manufacturing · doi:10.1016/j.rcim.2025.103076 · Cited by 70 · see citing works
Evaluation and Benchmarking of LLM Agents: A Survey
Mohammadi, M., Li, Y., Lo, J., & Yip, W. (2025) · Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.2 · doi:10.1145/3711896.3736570 · Cited by 74 · see citing works
“They struggle with evaluating open-ended or qualitative responses, such as natural language generation or creative problem-solving, where correctness is subjective.”
Self-citationLLM-Based Agents for Tool Learning: A Survey
Xu, W., Huang, C., Gao, S., & Shang, S. (2025) · Data Science and Engineering · doi:10.1007/s41019-025-00296-9 · Cited by 69 · see citing works
Pallagani, V., Muppasani, B. C., Roy, K., Fabiano, F., Loreggia, A., Murugesan, K., Srivastava, B., Rossi, F., Lior Horesh, C., & Sheth, A. (2024) · Proceedings of the International Conference on Automated Planning and Scheduling · doi:10.1609/icaps.v34i1.31503 · Cited by 62 · see citing works
Professional Certification Benchmark Dataset: The First 500 Jobs for Large Language Models
Noever, D. A., & Ciolino, M. (2023) · doi:10.5121/csit.2023.131211 · Cited by 38 · see citing works
Open-ti: open traffic intelligence with augmented language model
Da, L., Liou, K., Chen, T., Zhou, X. S., Luo, X., Yang, Y., & Wei, H. (2024) · International Journal of Machine Learning and Cybernetics · doi:10.1007/s13042-024-02190-8 · Cited by 38 · see citing works
PlanGenLLMs: A Modern Survey of LLM Planning Capabilities
Wei, H., Zhang, Z., He, S., Xia, T., Pan, S., & Liu, F. (2025) · Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2025.acl-long.958 · Cited by 36 · see citing works
CACTUS: Chemistry Agent Connecting Tool Usage to Science
McNaughton, A. D., Ramalaxmi, G. K. S., Kruel, A., Knutson, C. R., Varikoti, R. A., & Kumar, N. (2024) · ACS Omega · doi:10.1021/acsomega.4c08408 · Cited by 37 · see citing works
Automatic Evaluation of Attribution by Large Language Models
Yue, X., Wang, B., Chen, Z., Zhang, K., Su, Y., & Sun, H. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.307 · Cited by 62 · see citing works
Goldstein, S., & Kirk‐Giannini, C. D. (2025) · Asian Journal of Philosophy · doi:10.1007/s44204-025-00246-2 · Cited by 33 · see citing works
AgentNet: Decentralized Evolutionary Coordination for LLM-based Multi-Agent Systems
Yang, Y., Chai, H., Shao, S., Song, Y., Qi, S., Rui, R., & Zhang, W. (2025) · neural information processing systems · doi:10.52202/085713-3578 · Cited by 27 · see citing works
AgentBoard: An analytical evaluation board of multi-turn LLM agents
Ma, C., Zhang, J., Zhu, Z., Yang, C., Yang, Y., Jin, Y., Lan, Z., Kong, L., & He, J. (2024) · Advances in Neural Information Processing Systems 37 · doi:10.52202/079017-2365 · Cited by 26 · see citing works
Fan, H., Huang, J., Xu, J., Zhou, Y., Fuh, J. Y. H., Lu, W. F., & Li, B. (2025) · Materials & Design · doi:10.1016/j.matdes.2025.113644 · Cited by 24 · see citing works
Constraint-Consistent Skill Composition for Reliable Zero-Shot Task Generalization in LLM Agents
Meng, R., Huang, S., Zhang, X., Yang, Y., Zeng, K., & Huang, Z. (2026) · International Symposium on Big Data and Applied Statistics (ISBDAS) · doi:10.1109/isbdas69350.2026.11484408 · Cited by 21 · see citing works
Tool Learning in the Wild: Empowering Language Models as Automatic Tool Agents
Shi, Z., Gao, S., Yan, L., Feng, Y., Chen, X., Chen, Z., Yin, D., Verberne, S., & Ren, Z. (2025) · Proceedings of the ACM on Web Conference 2025 · doi:10.1145/3696410.3714825 · Cited by 21 · see citing works
Confucius: Iterative Tool Learning from Introspection Feedback by Easy-to-Difficult Curriculum
Gao, S., Shi, Z., Zhu, M., Fang, B., Xin, X., Ren, P., Chen, Z., Ma, J., & Ren, Z. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i16.29759 · Cited by 19 · see citing works
Large Language Model Evaluation Criteria Framework in Healthcare: Fuzzy MCDM Approach
Alabool, H. (2025) · SN Computer Science · doi:10.1007/s42979-024-03533-6 · Cited by 22 · see citing works
ModelScope-Agent: Building Your Customizable Agent System with Open-source Large Language Models
Li, C., Chen, H., Yan, M., Shen, W., Xu, H., Wu, Z., Zhang, Z., Zhou, W., Chen, Y., Chen, C., et al. (2023) · Conference on Empirical Methods in Natural Language Processing: System Demonstrations (EMNLP) · doi:10.18653/v1/2023.emnlp-demo.51 · Cited by 13 · see citing works
Self-citationGuoFeng, L. U. O., Weng, L., Li, Y., Sun, Y., Hong, Y., Wu, Y., Luo, R., Wang, L., Wang, C., & Chen, L. (2025) · IEEE Transactions on Mobile Computing · doi:10.1109/tmc.2025.3528413 · Cited by 13 · see citing works
Self-citationThe ethical evaluation of large language models and its optimization
Lyu, Y., & Du, Y. (2025) · AI and Ethics · doi:10.1007/s43681-024-00654-9 · Cited by 11 · see citing works
Evaluating LLM-based Agents for Multi-turn Conversations: A Survey
Guan, S., Wang, J., 渡辺, 将., Zhu, B., Lou, J., & Xiong, H. (2026) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3793671 · Cited by 28 · see citing works
A Taxonomy of Failures in Tool-Augmented LLMs
Winston, C., & Just, R. (2025) · International Conference/Workshop on Automation of Software Test · doi:10.1109/ast66626.2025.00019 · Cited by 10 · see citing works
Retrieval-Augmented Generation for Service Discovery: Chunking Strategies and Benchmarking
Pesl, R. D., Mathew, J. G., Mecella, M., & Aiello, M. (2026) · IEEE Transactions on Services Computing · doi:10.1109/tsc.2026.3665441 · Cited by 10 · see citing works
Towards Completeness-Oriented Tool Retrieval for Large Language Models
Qu, C., Dai, S., Wei, X., Cai, H., Wang, S., Yin, D., Xu, J., & Wen, J.-R. (2024) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3627673.3679847 · Cited by 15 · see citing works
Wang, R., Liang, S., Chen, Q., Zhang, J., & Qin, K. (2025) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3690624.3709238 · Cited by 9 · see citing works
Tool learning with language models: a comprehensive survey of methods, pipelines, and benchmarks
Chen, J., Wu, H., Pang, J., Wang, Y., Zhang, D., & Sun, C. (2025) · Vicinagearth. · doi:10.1007/s44336-025-00024-x · Cited by 11 · see citing works
AgentBoard: An Analytical Evaluation Board of Multi-turn LLM Agents
Ma, C., Zhang, J., Zhu, Z., Yang, C., Yang, Y., Jin, Y., Lan, Z., Kong, L., & He, J. (2024) · arXiv · doi:10.48550/arxiv.2401.13178 · Cited by 26 · see citing works
Seal-Tools: Self-instruct Tool Learning Dataset for Agent Tuning and Detailed Benchmark
Wu, M., Zhu, T., Han, H., Tan, C., Zhang, X., & Chen, W. (2024) · Lecture notes in computer science · doi:10.1007/978-981-97-9434-8_29 · Cited by 8 · see citing works
ScaleMCP: Dynamic and Auto-synchronizing Model Context Protocol Tools for LLM Agents
Lumer, E., Gulati, A., Subbiah, V. K., Basavaraju, P. H., & Burke, J. A. (2026) · Communications in computer and information science · doi:10.1007/978-3-032-15632-7_2 · Cited by 8 · see citing works
Kuan, C.-Y., Yang, C.-K., Huang, W., Lu, K.-H., & Lee, H.-Y. (2024) · IEEE Spoken Language Technology Workshop (SLT) · doi:10.1109/slt61566.2024.10832184 · Cited by 9 · see citing works
Chien, C. T., & Chien, C. H. (2025) · Applied Sciences · doi:10.3390/app151810123 · Cited by 7 · see citing works
Advanced System Integration: Analyzing OpenAPI Chunking for Retrieval-Augmented Generation
Pesl, R. D., Mathew, J. G., Mecella, M., & Aiello, M. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-94571-7_8 · Cited by 10 · see citing works
MetaTool Benchmark for Large Language Models: Deciding Whether to Use Tools and Which to Use
Huang, Y., Shi, J., Li, Y., Fan, C., Wu, S., Zhang, Q., Liu, Y., Zhou, P., Wan, Y., Gong, N. Z., et al. (2023) · arXiv · doi:10.48550/arxiv.2310.03128 · Cited by 7 · see citing works
Self-citationHan, Z., Wang, J.-X., Yan, X., Jiang, Z.-Y., Zhang, Y.-B., Liu, S., Gong, Q., & Song, C. (2025) · Applied Sciences · doi:10.3390/app15105663 · Cited by 8 · see citing works
Tu, S., Wang, Y., Yu, J., Xie, Y., Shi, Y., Wang, X., Zhang, J., Hou, L., & Li, J. (2024) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3637528.3671564 · Cited by 6 · see citing works
MAF: Multi-Aspect Feedback for Improving Reasoning in Large Language Models
Nathani, D., Wang, D., Pan, L., & Wang, W. Y. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.407 · Cited by 6 · see citing works
Action Engine: Automatic Workflow Generation in FaaS
Esashi, A., Lertpongrujikorn, P., Kato, S., & Salehi, M. A. (2025) · Future Generation Computer Systems · doi:10.1016/j.future.2025.107947 · Cited by 5 · see citing works
Koohestani, R., de Bekker, P., Koç, B., & Izadi, M. (2025) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2025.3644183 · Cited by 5 · see citing works
From benchmarks to deployment: a comprehensive review of agentic AI evaluation
Kehkashan, T., Abdullah, M. I., Al‐Shamayleh, A. S., Ivković, N., Ismail, N. A., Ahmad, S. S. S., Rehman, A., & Akhunzada, A. (2026) · Artificial Intelligence Review · doi:10.1007/s10462-026-11571-0 · Cited by 10 · see citing works
Go, H., & Park, S. (2025) · Scientific Reports · doi:10.1038/s41598-025-06469-w · Cited by 4 · see citing works
DeepAgent: A General Reasoning Agent with Scalable Toolsets
Li, X., Jiao, W., Jin, J., Dong, G., Jin, J., Wang, Y., Wang, H., Zhu, Y., Wen, J.-R., Yuan, L., et al. (2026) · International World Wide Web Conference (WWW) · doi:10.1145/3774904.3792460 · Cited by 22 · see citing works
Runtime Policy Enforcement for MCP-Based LLM Agents
Wang, S., Zhu, S., & Li, R. K. (2026) · Electronics · doi:10.3390/electronics15132829 · Cited by 7 · see citing works
UsmleGPT: An AI application for developing MCQs via multi-agent system
Jiang, Z., & Feng, S. (2025) · Software Impacts · doi:10.1016/j.simpa.2025.100742 · Cited by 4 · see citing works
Mathew, J. G., & Rossi, J. (2025) · Springer eBooks · doi:10.1007/978-3-031-92285-5_8 · Cited by 4 · see citing works
Gao, H.-A., Geng, J., Hua, W., Hu, M., Juan, X., Liu, H., Liu, S., Qiu, J., Qi, X., Wu, Y., et al. (2025) · arXiv · doi:10.48550/arxiv.2507.21046 · Cited by 3 · see citing works
SOLAR: Illuminating LLM performance in API discovery and service ranking for edge AI and IoT
Al‐Masri, E., & Subramanian, I. N. (2025) · Internet of Things · doi:10.1016/j.iot.2025.101630 · Cited by 3 · see citing works
ToolSQL: A Tool-Assisted Agent for SQL Verification and Refinement
Wang, Z., Zhang, R., Nie, Z., & Kim, J. (2025) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3711896.3737159 · Cited by 3 · see citing works
Causal Document-Grounded Dialogue Pre-training
Zhao, Y., Yu, B., Li, B., Yu, H., Li, J., Wang, C., Huang, F., Li, Y., & Zhang, N. L. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.443 · Cited by 4 · see citing works
Self-citationBig Loop and Atomization: A Holistic Review on the Expansion Capabilities of Large Language Models
Hu, Z., Huang, Y., Feng, J., & Deng, C. (2025) · Applied Sciences · doi:10.3390/app15179466 · Cited by 2 · see citing works
Can large language models exhibit cognitive and affective empathy as humans?
Yu, T., Pan, S., Fan, C., Luo, S., Jin, Y., & Zhao, B. (2025) · Computers in Human Behavior Artificial Humans · doi:10.1016/j.chbah.2025.100233 · Cited by 2 · see citing works
Enhanced tool invocation method through multi-model collaboration
Zhang, Y., Sun, H., Jia, M., Zhu, H., Zhang, D., & Li, X. (2025) · Vicinagearth. · doi:10.1007/s44336-025-00028-7 · Cited by 2 · see citing works
Investigating effective LLM-based in-context tool use: what matters and how to improve
Zheng, Y., Wei, H., Lu, J., Yin, L., Zhang, Y., Xu, C., Cui, H., Sun, T., Chen, S., & Qiu, X. (2026) · Frontiers of Computer Science · doi:10.1007/s11704-025-41365-6 · Cited by 2 · see citing works
ToolVQA: A Dataset for Multi-Step Reasoning VQA with External Tools
Yin, S., Lei, T., & Liu, Y. (2025) · IEEE International Conference on Computer Vision · doi:10.1109/iccv51701.2025.00421 · Cited by 18 · see citing works
TPE: Towards Better Compositional Reasoning over Cognitive Tools via Multi-persona Collaboration
Wang, H., Wang, H., Wang, L., Hu, M., Wang, R., Xue, B., Huang, Y., & Wong, K. (2024) · Lecture notes in computer science · doi:10.1007/978-981-97-9434-8_22 · Cited by 2 · see citing works
Can Large Language Models Exhibit Cognitive and Affective Empathy as Humans?Preprint
Pan, S., Fan, C., Zhao, B., Luo, S., & Jin, Y. (2024) · doi:10.31219/osf.io/w5rsu · Cited by 2 · see citing works
Clinical Agents Don’t CarePreprint
Klang, E., Glicksberg, B. S., Gorenshtein, A., Gavin, N. P., Freeman, R., Stump, L. S., Charney, A. W., Ting, D. S. W., Omar, M., & Nadkarni, G. N. (2025) · medRxiv · doi:10.1101/2025.10.17.25338226 · Cited by 2 · see citing works
TT-SI: Self-Improving LLM Agents with Test-Time Training
Acikgoz, E. C., Qian, C., Ji, H., Hakkani-Tür, D., & Tur, G. (2026) · Findings of the Association for Computational Linguistics: ACL 2026 · doi:10.18653/v1/2026.findings-acl.462 · Cited by 1 · see citing works
Search Arena: Analyzing Search-Augmented LLMs
Miroyan, M., Wu, T.-H., King, L., Li, T., Pan, J., Hu, X., Chiang, W.-L., Angelopoulos, A. N., Darrell, T., Norouzi, N., et al. (2025) · arXiv · doi:10.48550/arxiv.2506.05334 · Cited by 1 · see citing works
RTaC: A Generalized Framework for Tooling
Bhavsar, N., Thakur, A., Singh, A. L., & Patwa, A. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-65794-8_5 · Cited by 1 · see citing works
Tool Playgrounds: A Comprehensive and Analyzable Benchmark for LLM Tool Invocation
Dong, Z., Gong, R., Yang, Y., Wu, S., Yao, Y., Chen, S.-L., & Yin, X.-C. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10890828 · Cited by 1 · see citing works
Towards Large Language Model-Friendly APls
Wang, Y., Yang, Z., Wang, Z., Li, M., Wu, Z., & Chen, H. (2025) · ACM SIGOPS Operating Systems Review · doi:10.1145/3759441.3759445 · Cited by 1 · see citing works
Self-citationUser-Oriented Fuzzy Requirement Clarification for Agent Services Delivery Based on LLMs
Zhang, B., Chu, D., Xiong, S., Gong, Q., & Tu, Z. (2025) · IEEE Transactions on Services Computing · doi:10.1109/tsc.2025.3624915 · Cited by 1 · see citing works
RA-Gen: A Controllable Code Generation Framework Using ReAct for Multi-Agent Task Execution
Liu, A. Q., Li, H., Wang, B., Yang, A., & Li, H. (2025) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn64981.2025.11228706 · Cited by 1 · see citing works
Self-citationData-Driven Function Calling Improvements in Large Language Model for Online Financial QA
Tang, X. J., Chen, H., Li, S., Lyu, F., Shi, W., Li, L., Liu, D., Luo, W., Du, X., & He, X. (2026) · International World Wide Web Conference (WWW) · doi:10.1145/3774904.3792813 · Cited by 2 · see citing works
Tolis, D., Rytilahti, J., Weerakoon, O., Puhtila, P., Kaila, E., & Mäkilä, T. (2026) · Journal of Software Engineering Research and Development · doi:10.5753/jserd.2026.5969 · Cited by 1 · see citing works
Imaezue, G. C., Maram, K. V., Ajayi, D., Alohali, I., & Butta, R. K. (2026) · Journal of Speech, Language, and Hearing Research · doi:10.1044/2026_jslhr-26-00031 · Cited by 1 · see citing works
RS-Claw: Progressive Active Tool Exploration via Hierarchical Skill Trees for Remote Sensing Agents
Liu, L., Wang, Z., Li, Z., Ouyang, K., Tang, Z., Liu, C., Li, H., Yu, H., Yang, W., Yang, C., et al. (2026) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2026.3720894 · Cited by 1 · see citing works
Self-citationTrustworthiness Evaluation of Large Language Models
Chen, P., & Liu, S. (2025) · Springer eBooks · doi:10.1007/978-3-031-76770-8_12 · Cited by 2 · see citing works
LLM-powered Heterogeneous Information Network Analytics
Chen, H., Du, L., Chen, X., Ma, X., & Zhang, J. (2025) · International World Wide Web Conference (WWW) · doi:10.1145/3701716.3715450 · Cited by 1 · see citing works
AgentSHAP: Interpreting LLM Agent Tool Importance with Monte Carlo Shapley Value Estimation
Horovicz, M. (2025) · arXiv · doi:10.48550/arxiv.2512.12597
“We evaluate AgentSHAP on the API-Bank benchmark [4], which provides real executable tools and ground-truth annotations.”
VehicleMemBench: An Executable Benchmark for Multi-User Long-Term Memory in In-Vehicle Agents
Chen, Y., Xu, Y., Ding, X., Fang, X., Liu, S., Lin, L., Zhang, Q., Li, Y., Liu, Q., & Xu, T. (2026) · arXiv · doi:10.48550/arxiv.2603.23840
Self-citationLumer, E., Gulati, A., Subbiah, V. K., Basavaraju, P. H., & Burke, J. A. (2025) · arXiv · doi:10.48550/arxiv.2507.21428
AJ-Bench: Benchmarking Agent-as-a-Judge for Environment-Aware Evaluation
Shi, W., Wang, Y., Zhao, Y., Chen, Y., Feng, F., Hao, X., Su, X., Gu, Q., Su, H., Cai, X., et al. (2026) · arXiv · doi:10.48550/arxiv.2604.18240
Self-citationZhang, Y., Fan, X., Wang, J., Chen, C., Mo, F., Sakai, T., & Yamana, H. (2024) · Annual International ACM SIGIR Conference on Research and Development in Information Retrieval in the Asia Pacific Region (SIGIR-AP) · doi:10.1145/3673791.3698429
MapQaTor: A System for Efficient Annotation of Map Query DatasetsPreprint
Dihan, M. L., Ali, M. E., & Parvez, M. R. (2025) · Qeios · doi:10.32388/mi3wh4
Cooperative Agentic Framework for Enhanced Function Calling
Jiang, W., Ma, J., Zhang, Y., Wang, Y., Ji, W., & Zhang, Z. (2025) · Communications in computer and information science · doi:10.1007/978-981-96-4756-9_15
Imran, A., & Gopalakrishnan, K. (2025) · Apress eBooks · doi:10.1007/979-8-8688-0989-7_4
Deciding the Path: Leveraging Multi-Agent Systems for Solving Complex Tasks
Abbasnejad, I., Liu, X., & Roy, A. (2025) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw67362.2025.00405
Wang, C., Ding, H., Xing, Y., Zhao, B., Sui, D., & Tu, Z. (2025) · IEEE International Conference on Web Services (ICWS) · doi:10.1109/icws67624.2025.00070
Parallel Task Planning via Model Collaboration
Zhang, X., Zhu, T., Han, H., Pan, H., Wu, M., & Chen, W. (2025) · Lecture notes in computer science · doi:10.1007/978-981-95-3352-7_7
Virtual Assistant System in Banking Application With Fine-Tuned LLM and RAG Approach
Gunawan, E., Amadeo, A. J., Kamardi, C., & Suhartono, D. (2025) · International Conference on Informatics, Multimedia, Cyber and Information System (ICIMCIS) · doi:10.1109/icimcis68501.2025.11327124
Co-BioBench and SDTL: Advancing Human-AI Collaboration in Open-Ended Scientific Discovery
Liu, Y., Wang, X., & Zhang, Y. (2026) · Lecture notes in computer science · doi:10.1007/978-981-95-5764-6_24
PEARL: Plan Exploration and Adaptive Reinforcement Learning for Multihop Tool Use
Wang, Q., Lü, M., Wu, J., Hu, Y., & Liu, Y. (2026) · Lecture notes in computer science · doi:10.1007/978-981-95-7081-2_44
Tian, W. (2025) · International Conference on Machine Learning, Computational Intelligence and Pattern Recognition (MLCIPR) · doi:10.1109/mlcipr68329.2025.11407347
Arabic Prompts with English Tools: A Benchmark
Kubrak, K., El-Moselhy, A., Alsulami, A., Altuwaim, R., Fawaz, H. I., & Alsaby, F. A. (2025) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata66926.2025.11401864
WMAS: A Multi-Agent System Towards Intelligent and Customized Wireless Networks
Peng, J., Yuan, D., Ren, B., Fan, J., Wu, H., & Yang, L. (2025) · Global Communications Conference · doi:10.1109/globecom59602.2025.11432052
Integration of AI Content Generation-Enabled Virtual Museums into University History Education
Tan, S., Liu, Y., & Wang, L. (2026) · Applied System Innovation · doi:10.3390/asi9030064 · Cited by 1 · see citing works
Investigating the Impact of API Design Techniques for Tools in Agentic Applications
Bernardino, K. D., Queiroz, P. G. G., Guerra, E., Gonçalves, E., & Praxedes, L. B. F. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-39143-8_21
Yang, K., Liu, D., & Wang, Z. (2026) · Synthesis lectures on human language technologies · doi:10.1007/978-3-032-06264-2_10
ToolBox-RL: Learning to Generalize Tool Use Across Massive Repositories
Shi, X., Wang, R., Liu, H., & Li, P. (2026) · International World Wide Web Conference (WWW) · doi:10.1145/3774904.3792538
AI-Assisted REST API Design with Large Language Models
Martínez-Gil, J., Daxerer, C., Winterer, M., Neumüller, C., & Krump, M. (2026) · Lecture notes in business information processing · doi:10.1007/978-3-032-24216-7_5
- Authors
- Li, M., Zhao, Y., Yu, B., Song, F., Li, H., Yu, H., Li, Z., Huang, F., & Li, Y.
- Affiliations
- Alibaba Group (China) (CN)
- Hong Kong University of Science and Technology (HK)
- Peking University (CN)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2023
- Pages
- 3102-3116
- arXiv
- 2304.08244
- OpenAlex
- W4389518608
- Cited by
- 124
- References
- 27
- Citations by year
- 2026: 57 · 2025: 47 · 2024: 14 · 2023: 6
- FWCI
- 21.97
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- Rare & Special e-Zone (The Hong Kong University of Science and Technology) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- benchmark, error analysis
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
