The DOI matches its DataCite record; 33 of the 33 references it cites were found in a registry or an index.
API-Bank: A Comprehensive Benchmark for Tool-Augmented LLMs
Open accessAbstract
Recent research has demonstrated that Large Language Models (LLMs) can enhance their capabilities by utilizing external tools. However, three pivotal questions remain unanswered: (1) How effective are current LLMs in utilizing tools? (2) How can we enhance LLMs' ability to utilize tools? (3) What obstacles need to be overcome to leverage tools? To address these questions, we introduce API-Bank, a groundbreaking benchmark, specifically designed for tool-augmented LLMs. For the first question, we develop a runnable evaluation system consisting of 73 API tools. We annotate 314 tool-use dialogues with 753 API calls to assess the existing LLMs' capabilities in planning, retrieving, and calling APIs. For the second question, we construct a comprehensive training set containing 1,888 tool-use dialogues from 2,138 APIs spanning 1,000 distinct domains. Using this dataset, we train Lynx, a tool-augmented LLM initialized from Alpaca. Experimental results demonstrate that GPT-3.5 exhibits improved tool utilization compared to GPT-3, while GPT-4 excels in planning. However, there is still significant potential for further improvement. Moreover, Lynx surpasses Alpaca's tool utilization performance by more than 26 pts and approaches the effectiveness of GPT-3.5. Through error analysis, we highlight the key challenges for future research in this field to answer the third question.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source doesn’t support the claim as written.
In the citing paper
“It comprises a comprehensive Tool-Augmented LLM workflow, encompassing 53 commonly used API tools and 264 annotated dialogues, encompassing a total of 568 API calls.”
From A Survey on Evaluation of Large Language Models (2024) · cites this paper as [109]
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“As a result, LLM agents can directly build on the functions of the underlying model, allowing many of the evaluation techniques originally developed for LLMs to use tools [47].”
From Evaluation and Benchmarking of LLM Agents: A Survey (2025) · cites this paper as [47]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“the API-Bank benchmark [109] is introduced as the first benchmark explicitly designed for tool-augmented LLMs.”
From A Survey on Evaluation of Large Language Models (2024) · cites this paper as [109]
The cited source says what the claim attributes to it.
In the citing paper
“Others, such as FlowBench [96], ToolBench [38], and API-Bank [47], focus on tool use and function-calling across large API repositories.”
From Evaluation and Benchmarking of LLM Agents: A Survey (2025) · cites this paper as [47]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
LLaMA: Open and Efficient Foundation Language Models
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. (2023) · arXiv · doi:10.48550/arxiv.2302.13971 · Cited by 4,602
Sparks of Artificial General Intelligence: Early experiments with GPT-4
Bubeck, S., Chandrasekaran, V., Eldan, R., Gehrke, J., Horvitz, E., Kamar, E., Lee, P., Lee, Y. T., Li, Y., Lundberg, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.12712 · Cited by 1,526
Self-citationEvaluating Large Language Models Trained on Code
Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. D. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. (2021) · arXiv · doi:10.48550/arxiv.2107.03374 · Cited by 1,436
Toolformer: Language Models Can Teach Themselves to Use Tools
Schick, T., Dwivedi-Yu, J., Dessì, R., Raileanu, R., Lomeli, M., Zettlemoyer, L., Cancedda, N., & Scialom, T. (2023) · arXiv · doi:10.48550/arxiv.2302.04761 · Cited by 1,150
GLM: General Language Model Pretraining with Autoregressive Blank Infilling
Du, Z., Qian, Y., Liu, X., Ding, M., Qiu, J., Yang, Z., & Tang, J. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.26 · Cited by 1,068
Paleolithic Technology and Human Evolution
Ambrose, S. H. (2001) · Science · doi:10.1126/science.1059487 · Cited by 791
ReAct: Synergizing Reasoning and Acting in Language Models
Yao, S., Zhao, J., Yu, D., Du, N., Shafran, I., Narasimhan, K., & Cao, Y. (2022) · arXiv · doi:10.48550/arxiv.2210.03629 · Cited by 608
ToolQA: A Dataset for LLM Question Answering with External ToolsPreprint
Zhuang, Y.-C., Yu, Y., Wang, K., et al. (2023) · Neural Information Processing Systems · doi:10.48550/arxiv.2306.13304 · Cited by 421
ART: Automatic multi-step reasoning and tool-use for large language modelsPreprint
Paranjape, B., Lundberg, S., Singh, S., Hajishirzi, H., Zettlemoyer, L., & Ribeiro, M. T. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2303.09014 · Cited by 353
Research works printed but not linked: 1
- Cheng Qian, Chi Han, Yi R Fung, Yujia Qin, Zhiyuan Liu, and Heng Ji. 2023. Creator: Disentan- gling abstract and concrete reasonings of large lan- guage models through tool creation. arXiv preprint arXiv:2305.14318. · arXiv
Cited by
Sorted by citations
A Survey on Evaluation of Large Language Models
Chang, Y., Wang, X., Wang, J., Wu, Y., Yang, L., Zhu, K., Chen, H., Yi, X., Wang, C., Wang, Y., et al. (2024) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3641289 · Cited by 4,254
A survey on large language model based autonomous agents
Wang, L., Ma, C., Feng, X., Zhang, Z., Yang, H., Zhang, J., Chen, Z., Tang, J., Chen, X., Lin, Y., et al. (2024) · Frontiers of Computer Science · doi:10.1007/s11704-024-40231-1 · Cited by 1,941
Mind2Web: Towards a Generalist Agent for the Web
Deng, X., Gu, Y., Zheng, B., Chen, S., Stevens, S., Wang, B., Sun, H., & Su, Y. (2023) · arXiv · doi:10.48550/arxiv.2306.06070 · Cited by 134
Datasets for large language models: a comprehensive survey
Liu, Y., Cao, J., Liu, C., Ding, K., & Jin, L. (2025) · Artificial Intelligence Review · doi:10.1007/s10462-025-11403-7 · Cited by 268
Tool learning with large language models: a survey
Qu, C., Dai, S., Wei, X., Cai, H., Wang, S., Yin, D., Xu, J., & Wen, J.-R. (2025) · Frontiers of Computer Science · doi:10.1007/s11704-024-40678-2 · Cited by 266
Inadequacies of Large Language Model Benchmarks in the Era of Generative Artificial Intelligence
McIntosh, T. R., Susnjak, T., Arachchilage, N., Liu, T., Xu, D., Watters, P., & Halgamuge, M. N. (2026) · IEEE Transactions on Artificial Intelligence · doi:10.1109/tai.2025.3569516 · Cited by 104
RCAgent: Cloud Root Cause Analysis by Autonomous Agents with Tool-Augmented Large Language Models
Wang, Z., Liu, Z., Zhang, Y., Zhong, A., Wang, J., Yin, F., Fan, L., Wu, L., & Wen, Q. (2024) · Proceedings of the 33rd ACM International Conference on Information and Knowledge Management · doi:10.1145/3627673.3680016 · Cited by 99
Preference Ranking Optimization for Human Alignment
Song, F., Yu, B., Li, M., Yu, H., Huang, F., Li, Y., & Wang, H. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i17.29865 · Cited by 77
Self-citationQin, Y., Liang, S., Ye, Y., Zhu, K., Yan, L., Lu, Y., Lin, Y., Cong, X., Tang, X., Qian, B., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2307.16789 · Cited by 71
AIGC-empowered smart manufacturing: Prospects and challenges
Leng, J., Zheng, K., Li, R. C., Chen, C., Wang, B., Liu, Q., Chen, X., & Shen, W. (2025) · Robotics and Computer-Integrated Manufacturing · doi:10.1016/j.rcim.2025.103076 · Cited by 74
Evaluation and Benchmarking of LLM Agents: A Survey
Mohammadi, M., Li, Y., Lo, J., & Yip, W. (2025) · Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.2 · doi:10.1145/3711896.3736570 · Cited by 74
“They struggle with evaluating open-ended or qualitative responses, such as natural language generation or creative problem-solving, where correctness is subjective.”
Self-citationLLM-Based Agents for Tool Learning: A Survey
Xu, W., Huang, C., Gao, S., & Shang, S. (2025) · Data Science and Engineering · doi:10.1007/s41019-025-00296-9 · Cited by 87
Pallagani, V., Muppasani, B. C., Roy, K., Fabiano, F., Loreggia, A., Murugesan, K., Srivastava, B., Rossi, F., Lior Horesh, C., & Sheth, A. (2024) · Proceedings of the International Conference on Automated Planning and Scheduling · doi:10.1609/icaps.v34i1.31503 · Cited by 78
Professional Certification Benchmark Dataset: The First 500 Jobs for Large Language Models
Noever, D. A., & Ciolino, M. (2023) · doi:10.5121/csit.2023.131211 · Cited by 41
Open-ti: open traffic intelligence with augmented language model
Da, L., Liou, K., Chen, T., Zhou, X. S., Luo, X., Yang, Y., & Wei, H. (2024) · International Journal of Machine Learning and Cybernetics · doi:10.1007/s13042-024-02190-8 · Cited by 50
PlanGenLLMs: A Modern Survey of LLM Planning Capabilities
Wei, H., Zhang, Z., He, S., Xia, T., Pan, S., & Liu, F. (2025) · Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2025.acl-long.958 · Cited by 36
CACTUS: Chemistry Agent Connecting Tool Usage to Science
McNaughton, A. D., Ramalaxmi, G. K. S., Kruel, A., Knutson, C. R., Varikoti, R. A., & Kumar, N. (2024) · ACS Omega · doi:10.1021/acsomega.4c08408 · Cited by 53
Automatic Evaluation of Attribution by Large Language Models
Yue, X., Wang, B., Chen, Z., Zhang, K., Su, Y., & Sun, H. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.307 · Cited by 112
Goldstein, S., & Kirk‐Giannini, C. D. (2025) · Asian Journal of Philosophy · doi:10.1007/s44204-025-00246-2 · Cited by 44
AgentNet: Decentralized Evolutionary Coordination for LLM-based Multi-Agent Systems
Yang, Y., Chai, H., Shao, S., Song, Y., Qi, S., Rui, R., & Zhang, W. (2025) · neural information processing systems · doi:10.52202/085713-3578 · Cited by 118
AgentBoard: An analytical evaluation board of multi-turn LLM agents
Ma, C., Zhang, J., Zhu, Z., Yang, C., Yang, Y., Jin, Y., Lan, Z., Kong, L., & He, J. (2024) · Advances in Neural Information Processing Systems 37 · doi:10.52202/079017-2365 · Cited by 26
Survey on Evaluation of LLM-based Agents
Yehudai, A., Eden, L., Li, A., Uziel, G., Zhao, Y., Bar-Haim, R., Cohan, A., & Shmueli-Scheuer, M. (2025) · arXiv · doi:10.48550/arxiv.2503.16416 · Cited by 26
Self-citationFan, H., Huang, J., Xu, J., Zhou, Y., Fuh, J. Y. H., Lu, W. F., & Li, B. (2025) · Materials & Design · doi:10.1016/j.matdes.2025.113644 · Cited by 26
Constraint-Consistent Skill Composition for Reliable Zero-Shot Task Generalization in LLM Agents
Meng, R., Huang, S., Zhang, X., Yang, Y., Zeng, K., & Huang, Z. (2026) · International Symposium on Big Data and Applied Statistics (ISBDAS) · doi:10.1109/isbdas69350.2026.11484408 · Cited by 26
Tool Learning in the Wild: Empowering Language Models as Automatic Tool Agents
Shi, Z., Gao, S., Yan, L., Feng, Y., Chen, X., Chen, Z., Yin, D., Verberne, S., & Ren, Z. (2025) · Proceedings of the ACM on Web Conference 2025 · doi:10.1145/3696410.3714825 · Cited by 21
Confucius: Iterative Tool Learning from Introspection Feedback by Easy-to-Difficult Curriculum
Gao, S., Shi, Z., Zhu, M., Fang, B., Xin, X., Ren, P., Chen, Z., Ma, J., & Ren, Z. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i16.29759 · Cited by 31
Large Language Model Evaluation Criteria Framework in Healthcare: Fuzzy MCDM Approach
Alabool, H. M. (2025) · SN Computer Science · doi:10.1007/s42979-024-03533-6 · Cited by 22
ModelScope-Agent: Building Your Customizable Agent System with Open-source Large Language Models
Li, C., Chen, H., Yan, M., Shen, W., Xu, H., Wu, Z., Zhang, Z., Zhou, W., Chen, Y., Chen, C., et al. (2023) · Conference on Empirical Methods in Natural Language Processing: System Demonstrations (EMNLP) · doi:10.18653/v1/2023.emnlp-demo.51 · Cited by 14
Self-citationGuoFeng, L. U. O., Weng, L., Li, Y., Sun, Y., Hong, Y., Wu, Y., Luo, R., Wang, L., Wang, C., & Chen, L. (2025) · IEEE Transactions on Mobile Computing · doi:10.1109/tmc.2025.3528413 · Cited by 14
Self-citationAn LLM Compiler for Parallel Function Calling
Kim, S., Moon, S., Tabrizi, R., Lee, N., Mahoney, M. W., Keutzer, K., & Gholami, A. (2023) · arXiv · doi:10.48550/arxiv.2312.04511 · Cited by 12
“Second, LLMCompiler supports dynamic replanning which is important for problems whose execution flow cannot be determined statically in the beginning (Sec.”
The ethical evaluation of large language models and its optimization
Lyu, Y., & Du, Y. (2025) · AI and Ethics · doi:10.1007/s43681-024-00654-9 · Cited by 11
Evaluating LLM-based Agents for Multi-turn Conversations: A Survey
Guan, S., Wang, J., 渡辺, 将., Zhu, B., Lou, J., & Xiong, H. (2026) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3793671 · Cited by 71
A Taxonomy of Failures in Tool-Augmented LLMs
Winston, C., & Just, R. (2025) · International Conference/Workshop on Automation of Software Test · doi:10.1109/ast66626.2025.00019 · Cited by 10
Retrieval-Augmented Generation for Service Discovery: Chunking Strategies and Benchmarking
Pesl, R. D., Mathew, J. G., Mecella, M., & Aiello, M. (2026) · IEEE Transactions on Services Computing · doi:10.1109/tsc.2026.3665441 · Cited by 13
Towards Completeness-Oriented Tool Retrieval for Large Language Models
Qu, C., Dai, S., Wei, X., Cai, H., Wang, S., Yin, D., Xu, J., & Wen, J.-R. (2024) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3627673.3679847 · Cited by 36
Wang, R., Liang, S., Chen, Q., Zhang, J., & Qin, K. (2025) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3690624.3709238 · Cited by 9
Tool learning with language models: a comprehensive survey of methods, pipelines, and benchmarks
Chen, J., Wu, H., Pang, J., Wang, Y., Zhang, D., & Sun, C. (2025) · Vicinagearth. · doi:10.1007/s44336-025-00024-x · Cited by 15
AgentBoard: An Analytical Evaluation Board of Multi-turn LLM Agents
Ma, C., Zhang, J., Zhu, Z., Yang, C., Yang, Y., Jin, Y., Lan, Z., Kong, L., & He, J. (2024) · arXiv · doi:10.48550/arxiv.2401.13178 · Cited by 26
Seal-Tools: Self-instruct Tool Learning Dataset for Agent Tuning and Detailed Benchmark
Wu, M., Zhu, T., Han, H., Tan, C., Zhang, X., & Chen, W. (2024) · Lecture notes in computer science · doi:10.1007/978-981-97-9434-8_29 · Cited by 36
ScaleMCP: Dynamic and Auto-synchronizing Model Context Protocol Tools for LLM Agents
Lumer, E., Gulati, A., Subbiah, V. K., Basavaraju, P. H., & Burke, J. A. (2026) · Communications in computer and information science · doi:10.1007/978-3-032-15632-7_2 · Cited by 8
Kuan, C.-Y., Yang, C.-K., Huang, W., Lu, K.-H., & Lee, H.-Y. (2024) · IEEE Spoken Language Technology Workshop (SLT) · doi:10.1109/slt61566.2024.10832184 · Cited by 13
Chien, C. T., & Chien, C. H. (2025) · Applied Sciences · doi:10.3390/app151810123 · Cited by 7
Advanced System Integration: Analyzing OpenAPI Chunking for Retrieval-Augmented Generation
Pesl, R. D., Mathew, J. G., Mecella, M., & Aiello, M. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-94571-7_8 · Cited by 12
MetaTool Benchmark for Large Language Models: Deciding Whether to Use Tools and Which to Use
Huang, Y., Shi, J., Li, Y., Fan, C., Wu, S., Zhang, Q., Liu, Y., Zhou, P., Wan, Y., Gong, N. Z., et al. (2023) · arXiv · doi:10.48550/arxiv.2310.03128 · Cited by 7
Self-citationHan, Z., Wang, J., Yan, X., Jiang, Z., Zhang, Y., Liu, S., Gong, Q., & Song, C. (2025) · Applied Sciences · doi:10.3390/app15105663 · Cited by 7
Tu, S., Wang, Y., Yu, J., Xie, Y., Shi, Y., Wang, X., Zhang, J., Hou, L., & Li, J. (2024) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3637528.3671564 · Cited by 7
MAF: Multi-Aspect Feedback for Improving Reasoning in Large Language Models
Nathani, D., Wang, D., Pan, L., & Wang, W. Y. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.407 · Cited by 13
Action Engine: Automatic Workflow Generation in FaaS
Esashi, A., Lertpongrujikorn, P., Kato, S., & Salehi, M. A. (2025) · Future Generation Computer Systems · doi:10.1016/j.future.2025.107947 · Cited by 7
Koohestani, R., de Bekker, P., Koç, B., & Izadi, M. (2025) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2025.3644183 · Cited by 13
From benchmarks to deployment: a comprehensive review of agentic AI evaluation
Kehkashan, T., Abdullah, M. I., Al‐Shamayleh, A. S., Ivković, N., Ismail, N. A., Ahmad, S. S. S., Rehman, A., & Akhunzada, A. (2026) · Artificial Intelligence Review · doi:10.1007/s10462-026-11571-0 · Cited by 14
Go, H., & Park, S. (2025) · Scientific Reports · doi:10.1038/s41598-025-06469-w · Cited by 5
DeepAgent: A General Reasoning Agent with Scalable Toolsets
Li, X., Jiao, W., Jin, J., Dong, G., Jin, J., Wang, Y., Wang, H., Zhu, Y., Wen, J.-R., Yuan, L., et al. (2026) · International World Wide Web Conference (WWW) · doi:10.1145/3774904.3792460 · Cited by 74
Runtime Policy Enforcement for MCP-Based LLM Agents
Wang, S., Zhu, S., & Li, R. K. (2026) · Electronics · doi:10.3390/electronics15132829 · Cited by 10
UsmleGPT: An AI application for developing MCQs via multi-agent system
Jiang, Z., & Feng, S. (2025) · Software Impacts · doi:10.1016/j.simpa.2025.100742 · Cited by 6
Mathew, J. G., & Rossi, J. (2025) · Springer eBooks · doi:10.1007/978-3-031-92285-5_8 · Cited by 4
Gao, H.-A., Geng, J., Hua, W., Hu, M., Juan, X., Liu, H., Liu, S., Qiu, J., Qi, X., Wu, Y., et al. (2025) · arXiv · doi:10.48550/arxiv.2507.21046 · Cited by 3
SOLAR: Illuminating LLM performance in API discovery and service ranking for edge AI and IoT
Al‐Masri, E., & Subramanian, I. N. (2025) · Internet of Things · doi:10.1016/j.iot.2025.101630 · Cited by 3
ToolSQL: A Tool-Assisted Agent for SQL Verification and Refinement
Wang, Z., Zhang, R., Nie, Z., & Kim, J. (2025) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3711896.3737159 · Cited by 3
Causal Document-Grounded Dialogue Pre-training
Zhao, Y., Yu, B., Li, B., Yu, H., Li, J., Wang, C., Huang, F., Li, Y., & Zhang, N. L. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.443 · Cited by 10
Self-citationBig Loop and Atomization: A Holistic Review on the Expansion Capabilities of Large Language Models
Hu, Z., Huang, Y., Feng, J., & Deng, C. (2025) · Applied Sciences · doi:10.3390/app15179466 · Cited by 3
Can large language models exhibit cognitive and affective empathy as humans?
Yu, T., Pan, S., Fan, C., Luo, S., Jin, Y., & Zhao, B. (2025) · Computers in Human Behavior Artificial Humans · doi:10.1016/j.chbah.2025.100233 · Cited by 9
Enhanced tool invocation method through multi-model collaboration
Zhang, Y., Sun, H., Jia, M., Zhu, H., Zhang, D., & Li, X. (2025) · Vicinagearth. · doi:10.1007/s44336-025-00028-7 · Cited by 2
Investigating effective LLM-based in-context tool use: what matters and how to improve
Zheng, Y., Wei, H., Lu, J., Yin, L., Zhang, Y., Xu, C., Cui, H., Sun, T., Chen, S., & Qiu, X. (2026) · Frontiers of Computer Science · doi:10.1007/s11704-025-41365-6 · Cited by 2
ToolVQA: A Dataset for Multi-Step Reasoning VQA with External Tools
Yin, S., Lei, T., & Liu, Y. (2025) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51701.2025.00421 · Cited by 5
TPE: Towards Better Compositional Reasoning over Cognitive Tools via Multi-persona Collaboration
Wang, H., Wang, H., Wang, L., Hu, M., Wang, R., Xue, B., Huang, Y., & Wong, K. (2024) · Lecture notes in computer science · doi:10.1007/978-981-97-9434-8_22 · Cited by 2
Can Large Language Models Exhibit Cognitive and Affective Empathy as Humans?Preprint
Pan, S., Fan, C., Zhao, B., Luo, S., & Jin, Y. (2024) · doi:10.31219/osf.io/w5rsu · Cited by 2
Clinical Agents Don’t CarePreprint
Klang, E., Glicksberg, B. S., Gorenshtein, A., Gavin, N. P., Freeman, R., Stump, L. S., Charney, A. W., Ting, D. S. W., Omar, M., & Nadkarni, G. N. (2025) · medRxiv · doi:10.1101/2025.10.17.25338226 · Cited by 2
Memento: Fine-tuning LLM Agents without Fine-tuning LLMs
Zhou, H., Chen, Y., Guo, S., Yan, X., Lee, K. H., Wang, Z., Lee, K. Y., Zhang, G., Shao, K., Yang, L., et al. (2025) · arXiv · doi:10.48550/arxiv.2508.16153 · Cited by 2
TT-SI: Self-Improving LLM Agents with Test-Time Training
Acikgoz, E. C., Qian, C., Ji, H., Hakkani-Tür, D., & Tur, G. (2026) · Findings of the Association for Computational Linguistics: ACL 2026 · doi:10.18653/v1/2026.findings-acl.462 · Cited by 1
Search Arena: Analyzing Search-Augmented LLMs
Miroyan, M., Wu, T.-H., King, L., Li, T., Pan, J., Hu, X., Chiang, W.-L., Angelopoulos, A. N., Darrell, T., Norouzi, N., et al. (2025) · arXiv · doi:10.48550/arxiv.2506.05334 · Cited by 1
RTaC: A Generalized Framework for Tooling
Bhavsar, N., Thakur, A., Singh, A. L., & Patwa, A. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-65794-8_5 · Cited by 1
Tool Playgrounds: A Comprehensive and Analyzable Benchmark for LLM Tool Invocation
Dong, Z., Gong, R., Yang, Y., Wu, S., Yao, Y., Chen, S.-L., & Yin, X.-C. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10890828 · Cited by 2
Towards Large Language Model-Friendly APls
Wang, Y., Yang, Z., Wang, Z., Li, M., Wu, Z., & Chen, H. (2025) · ACM SIGOPS Operating Systems Review · doi:10.1145/3759441.3759445 · Cited by 1
Self-citationUser-Oriented Fuzzy Requirement Clarification for Agent Services Delivery Based on LLMs
Zhang, B., Chu, D., Xiong, S., Gong, Q., & Tu, Z. (2025) · IEEE Transactions on Services Computing · doi:10.1109/tsc.2025.3624915 · Cited by 1
RA-Gen: A Controllable Code Generation Framework Using ReAct for Multi-Agent Task Execution
Liu, A. Q., Li, H., Wang, B., Yang, A., & Li, H. (2025) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn64981.2025.11228706 · Cited by 1
Self-citationData-Driven Function Calling Improvements in Large Language Model for Online Financial QA
Tang, X. J., Chen, H., Li, S., Lyu, F., Shi, W., Li, L., Liu, D., Luo, W., Du, X., & He, X. (2026) · International World Wide Web Conference (WWW) · doi:10.1145/3774904.3792813 · Cited by 2
Tolis, D., Rytilahti, J., Weerakoon, O., Puhtila, P., Kaila, E., & Mäkilä, T. (2026) · Journal of Software Engineering Research and Development · doi:10.5753/jserd.2026.5969 · Cited by 1
Imaezue, G. C., Maram, K. V., Ajayi, D., Alohali, I., & Butta, R. K. (2026) · Journal of Speech, Language, and Hearing Research · doi:10.1044/2026_jslhr-26-00031 · Cited by 1
RS-Claw: Progressive Active Tool Exploration via Hierarchical Skill Trees for Remote Sensing Agents
Liu, L., Wang, Z., Li, Z., Ouyang, K., Tang, Z., Liu, C., Li, H., Yu, H., Yang, W., Yang, C., et al. (2026) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2026.3720894 · Cited by 2
Self-citationTrustworthiness Evaluation of Large Language Models
Chen, P., & Liu, S. (2025) · Springer eBooks · doi:10.1007/978-3-031-76770-8_12 · Cited by 2
LLM-powered Heterogeneous Information Network Analytics
Chen, H., Du, L., Chen, X., Ma, X., & Zhang, J. (2025) · International World Wide Web Conference (WWW) · doi:10.1145/3701716.3715450 · Cited by 3
AgentSHAP: Interpreting LLM Agent Tool Importance with Monte Carlo Shapley Value Estimation
Horovicz, M. (2025) · arXiv · doi:10.48550/arxiv.2512.12597
“We evaluate AgentSHAP on the API-Bank benchmark [4], which provides real executable tools and ground-truth annotations.”
VehicleMemBench: An Executable Benchmark for Multi-User Long-Term Memory in In-Vehicle Agents
Chen, Y., Xu, Y., Ding, X., Fang, X., Liu, S., Lin, L., Zhang, Q., Li, Y., Liu, Q., & Xu, T. (2026) · arXiv · doi:10.48550/arxiv.2603.23840
Self-citationLumer, E., Gulati, A., Subbiah, V. K., Basavaraju, P. H., & Burke, J. A. (2025) · arXiv · doi:10.48550/arxiv.2507.21428 · Cited by 19
AJ-Bench: Benchmarking Agent-as-a-Judge for Environment-Aware Evaluation
Shi, W., Wang, Y., Zhao, Y., Chen, Y., Feng, F., Hao, X., Su, X., Gu, Q., Su, H., Cai, X., et al. (2026) · arXiv · doi:10.48550/arxiv.2604.18240
Self-citationZhang, Y., Fan, X., Wang, J., Chen, C., Mo, F., Sakai, T., & Yamana, H. (2024) · Annual International ACM SIGIR Conference on Research and Development in Information Retrieval in the Asia Pacific Region (SIGIR-AP) · doi:10.1145/3673791.3698429
MapQaTor: A System for Efficient Annotation of Map Query DatasetsPreprint
Dihan, M. L., Ali, M. E., & Parvez, M. R. (2025) · Qeios · doi:10.32388/mi3wh4
Cooperative Agentic Framework for Enhanced Function Calling
Jiang, W., Ma, J., Zhang, Y., Wang, Y., Ji, W., & Zhang, Z. (2025) · Communications in computer and information science · doi:10.1007/978-981-96-4756-9_15
Imran, A., & Gopalakrishnan, K. (2025) · Apress eBooks · doi:10.1007/979-8-8688-0989-7_4
Deciding the Path: Leveraging Multi-Agent Systems for Solving Complex Tasks
Abbasnejad, I., Liu, X., & Roy, A. (2025) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw67362.2025.00405
Wang, C., Ding, H., Xing, Y., Zhao, B., Sui, D., & Tu, Z. (2025) · IEEE International Conference on Web Services (ICWS) · doi:10.1109/icws67624.2025.00070
Parallel Task Planning via Model Collaboration
Zhang, X., Zhu, T., Han, H., Pan, H., Wu, M., & Chen, W. (2025) · Lecture notes in computer science · doi:10.1007/978-981-95-3352-7_7
Virtual Assistant System in Banking Application With Fine-Tuned LLM and RAG Approach
Gunawan, E., Amadeo, A. J., Kamardi, C., & Suhartono, D. (2025) · International Conference on Informatics, Multimedia, Cyber and Information System (ICIMCIS) · doi:10.1109/icimcis68501.2025.11327124 · Cited by 1
Co-BioBench and SDTL: Advancing Human-AI Collaboration in Open-Ended Scientific Discovery
Liu, Y., Wang, X., & Zhang, Y. (2026) · Lecture notes in computer science · doi:10.1007/978-981-95-5764-6_24
PEARL: Plan Exploration and Adaptive Reinforcement Learning for Multihop Tool Use
Wang, Q., Lü, M., Wu, J., Hu, Y., & Liu, Y. (2026) · Lecture notes in computer science · doi:10.1007/978-981-95-7081-2_44
Tian, W. (2025) · International Conference on Machine Learning, Computational Intelligence and Pattern Recognition (MLCIPR) · doi:10.1109/mlcipr68329.2025.11407347 · Cited by 1
Arabic Prompts with English Tools: A Benchmark
Kubrak, K., El-Moselhy, A., Alsulami, A., Altuwaim, R., Fawaz, H. I., & Alsaby, F. A. (2025) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata66926.2025.11401864
WMAS: A Multi-Agent System Towards Intelligent and Customized Wireless Networks
Peng, J., Yuan, D., Ren, B., Fan, J., Wu, Z., & Yang, L. (2025) · Globecom · doi:10.1109/globecom59602.2025.11432052
Integration of AI Content Generation-Enabled Virtual Museums into University History Education
Tan, S., Liu, Y., & Wang, L. (2026) · Applied System Innovation · doi:10.3390/asi9030064 · Cited by 1
Investigating the Impact of API Design Techniques for Tools in Agentic Applications
Bernardino, K. D., Queiroz, P. G. G., Guerra, E., Gonçalves, E., & Praxedes, L. B. F. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-39143-8_21
Metadata
- Authors
- Li, M., Zhao, Y., Yu, B., Song, F., Li, H., Yu, H., Li, Z., Huang, F., & Li, Y.
- Affiliations
- Alibaba Group (China) (CN)
- Hong Kong University of Science and Technology (HK)
- Peking University (CN)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2023
- Pages
- 3102-3116
- arXiv
- 2304.08244
- OpenAlex
- W4389518608
- Cited by
- 124
- References
- 27
- Citations by year
- 2026: 57 · 2025: 47 · 2024: 14 · 2023: 6
- FWCI
- 21.97
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- Rare & Special e-Zone (The Hong Kong University of Science and Technology) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- benchmark, error analysis
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
