The DOI matches its DataCite record; 33 of the 33 references it cites were found in a registry or an index.
API-Bank: A Comprehensive Benchmark for Tool-Augmented LLMs
Open accessAbstract
Recent research has demonstrated that Large Language Models (LLMs) can enhance their capabilities by utilizing external tools. However, three pivotal questions remain unanswered: (1) How effective are current LLMs in utilizing tools? (2) How can we enhance LLMs' ability to utilize tools? (3) What obstacles need to be overcome to leverage tools? To address these questions, we introduce API-Bank, a groundbreaking benchmark, specifically designed for tool-augmented LLMs. For the first question, we develop a runnable evaluation system consisting of 73 API tools. We annotate 314 tool-use dialogues with 753 API calls to assess the existing LLMs' capabilities in planning, retrieving, and calling APIs. For the second question, we construct a comprehensive training set containing 1,888 tool-use dialogues from 2,138 APIs spanning 1,000 distinct domains. Using this dataset, we train Lynx, a tool-augmented LLM initialized from Alpaca. Experimental results demonstrate that GPT-3.5 exhibits improved tool utilization compared to GPT-3, while GPT-4 excels in planning. However, there is still significant potential for further improvement. Moreover, Lynx surpasses Alpaca's tool utilization performance by more than 26 pts and approaches the effectiveness of GPT-3.5. Through error analysis, we highlight the key challenges for future research in this field to answer the third question.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source doesn’t support the claim as written.
In the citing paper
“It comprises a comprehensive Tool-Augmented LLM workflow, encompassing 53 commonly used API tools and 264 annotated dialogues, encompassing a total of 568 API calls.”
From A Survey on Evaluation of Large Language Models (2024) · cites this paper as [109]
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“As a result, LLM agents can directly build on the functions of the underlying model, allowing many of the evaluation techniques originally developed for LLMs to use tools [47].”
From Evaluation and Benchmarking of LLM Agents: A Survey (2025) · cites this paper as [47]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“the API-Bank benchmark [109] is introduced as the first benchmark explicitly designed for tool-augmented LLMs.”
From A Survey on Evaluation of Large Language Models (2024) · cites this paper as [109]
The cited source says what the claim attributes to it.
In the citing paper
“Others, such as FlowBench [96], ToolBench [38], and API-Bank [47], focus on tool use and function-calling across large API repositories.”
From Evaluation and Benchmarking of LLM Agents: A Survey (2025) · cites this paper as [47]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
LLaMA: Open and Efficient Foundation Language Models
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. (2023) · arXiv · doi:10.48550/arxiv.2302.13971 · Cited by 4,602
Sparks of Artificial General Intelligence: Early experiments with GPT-4
Bubeck, S., Chandrasekaran, V., Eldan, R., Gehrke, J., Horvitz, E., Kamar, E., Lee, P., Lee, Y. T., Li, Y., Lundberg, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.12712 · Cited by 1,526
Self-citationEvaluating Large Language Models Trained on Code
Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. D. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. (2021) · arXiv · doi:10.48550/arxiv.2107.03374 · Cited by 1,436
Toolformer: Language Models Can Teach Themselves to Use Tools
Schick, T., Dwivedi-Yu, J., Dessì, R., Raileanu, R., Lomeli, M., Zettlemoyer, L., Cancedda, N., & Scialom, T. (2023) · arXiv · doi:10.48550/arxiv.2302.04761 · Cited by 1,150
GLM: General Language Model Pretraining with Autoregressive Blank Infilling
Du, Z., Qian, Y., Liu, X., Ding, M., Qiu, J., Yang, Z., & Tang, J. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.26 · Cited by 1,068
Paleolithic Technology and Human Evolution
Ambrose, S. H. (2001) · Science · doi:10.1126/science.1059487 · Cited by 791
ReAct: Synergizing Reasoning and Acting in Language Models
Yao, S., Zhao, J., Yu, D., Du, N., Shafran, I., Narasimhan, K., & Cao, Y. (2022) · arXiv · doi:10.48550/arxiv.2210.03629 · Cited by 608
ToolQA: A Dataset for LLM Question Answering with External ToolsPreprint
Zhuang, Y.-C., Yu, Y., Wang, K., et al. (2023) · Neural Information Processing Systems · doi:10.48550/arxiv.2306.13304 · Cited by 421
ART: Automatic multi-step reasoning and tool-use for large language modelsPreprint
Paranjape, B., Lundberg, S., Singh, S., Hajishirzi, H., Zettlemoyer, L., & Ribeiro, M. T. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2303.09014 · Cited by 353
Research works printed but not linked: 1
- Cheng Qian, Chi Han, Yi R Fung, Yujia Qin, Zhiyuan Liu, and Heng Ji. 2023. Creator: Disentan- gling abstract and concrete reasonings of large lan- guage models through tool creation. arXiv preprint arXiv:2305.14318. · arXiv
Cited by
Sorted by citations
Items 101–144 of 144
Yang, K., Liu, D., & Wang, Z. (2026) · Synthesis lectures on human language technologies · doi:10.1007/978-3-032-06264-2_10
ToolBox-RL: Learning to Generalize Tool Use Across Massive Repositories
Shi, X., Wang, R., Liu, H., & Li, P. (2026) · International World Wide Web Conference (WWW) · doi:10.1145/3774904.3792538
AI-Assisted REST API Design with Large Language Models
Martínez-Gil, J., Daxerer, C., Winterer, M., Neumüller, C., & Krump, M. (2026) · Lecture notes in business information processing · doi:10.1007/978-3-032-24216-7_5
ReTools: Reflection-Enhanced Tool Invocation for Domain-Specific QA
Dong, F., Zhou, T., Zhang, M., Pan, X., Zhao, D., Xu, L., Wen, Y., & Yuan, X. (2026) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp55912.2026.11463575
KG-ToolPlan: Knowledge Graph-Guided Reasoning for Efficient LLM Tool Selection
Cheng, Y., Wan, Z., Li, C., Wang, S., Ma, T., & Zhao, W. (2026) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp55912.2026.11461723
Li, H., Feng, L., & Wang, D. (2026) · Electronics · doi:10.3390/electronics15102011
Self-citationExp-ToolPlan: Contrastive Experience Learning for Similar Tool Selection
Cheng, Y., Yan, M., & Wan, Z. (2026) · International Conference on Generative Artificial Intelligence and Information Security (GAIIS) · doi:10.1109/gaiis69281.2026.11519215
SVI-Bench: A Dynamic Microworld for Strategic Video Intelligence
Pan, Y., Han, Y., Ha, S., Islam, M. M., Zhang, B., Torresani, L., & Bertasius, G. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-37095-2_12
Luo, Y. (2026) · Research Square · doi:10.21203/rs.3.rs-9663460/v1
Lin, Z., Chen, S., & Sun, H. (2026) · ACM Symposium on Access Control Models and Technologies (SACMAT) · doi:10.1145/3750555.3811883
RAP: Resource-Adaptive Planning for Efficient LLM Tool Calling in Edge Computing
Liu, T., Tang, Z., Guo, J., Lou, J., Wang, T., & Weijia Jia, B. (2026) · IEEE Conference on Computer Communications (INFOCOM) · doi:10.1109/infocom59046.2026.11571461 · Cited by 1
Esiur: A Resource-Centric Distributed Runtime Framework for Real-Time State Synchronization
Zamil, A. K., & Jasim, A. D. (2026) · IEEE Access · doi:10.1109/access.2026.3711683
VTool: Benchmarking Tool Use Ability in Multi-modal Context
Li, Y., Li, Z., Feng, T., Deng, Y., & ZHANG, Y. (2026) · Lecture notes in computer science · doi:10.1007/978-981-92-2852-2_48
Self-citationMa, J., Wang, C., Zhang, Y., Wang, Q., Yang, W., & Zhao, Z. (2026) · Lecture notes in computer science · doi:10.1007/978-981-92-3403-5_9
MGFS: Structure-Aware Multi-granular Function Selection for LLM-Based Agents
Zhang, Y., Li, C., Ma, J., Wang, Q., Yang, W., & Zhao, Z. (2026) · Lecture notes in computer science · doi:10.1007/978-981-92-3403-5_11
Vicente-Martínez, P., Ferrer-Baixauli, C., Soria‐Olivas, E., Fernández-Baldera, A., García-Escrivà, M. Á., & William, E. (2026) · Applied Sciences · doi:10.3390/app16178890
Perera, K. J. P. G., Nanayakkara, S., & Huff, E. W. (2026) · ACM Conference on Conversational User Interfaces (CUI) · doi:10.1145/3816046.3816225
Wang, Z., Hu, X., Li, P., Di, K., Jiang, Y., Zuo, T., Wang, Y., Shan, D., & Liu, T. (2026) · Service Oriented Computing and Applications · doi:10.1007/s11761-026-00502-0
Conflict-Aware Evidence Arbitration for Risk-Constrained Energy Saving in Radio Access NetworkPreprint
Jiang, L., Wang, Z., Qin, Z., Xie, B., & Zeng, H. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.7197341
Conflict-Aware Evidence Arbitration for Risk-Constrained Energy Saving in Radio Access NetworkPreprint
Jiang, L., Wang, Z., Qin, Z., Xie, B., & Zeng, H. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.7201208
Vatsal, S., Dubey, H., & Bajaj, A. (2026) · IEEE Access · doi:10.1109/access.2026.3720396
TopoMLP++: Toward Strong and Scalable Lane Topology Reasoning in Autonomous Driving
Wu, D., Han, W., Xu, C., & Shen, J. (2026) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2026.3721163
FactorMiner: A Self-Evolving Agent with Skills and Experience Memory for Financial Alpha Discovery
Wang, Y., Xu, J., Zhang, H., Huang, S., Sun, D., & Zhang, X.-P. S. (2026) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3770855.3818978 · Cited by 19
PAUSE: A User-Centric Benchmark for Personal AI Assistants in Unified Service Environments
Chen, H., Shi, X., Wang, Y., Chen, J., & Niu, D. (2026) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3770855.3817565
Pysklo, H. M., Zhuravel, A., & Watson, P. D. (2026) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3770855.3817555 · Cited by 2
Yan, J., Fang, X., Zhu, J., Wu, C., Yang, Y., Du, C., Fang, M., Zhang, K., Zhang, Z., & Liu, Q. (2026) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3770855.3817556
Boundary-aware tool routing framework for large language models in vertical domains scenarios
Zhang, H., Shi, X., Chen, H., Zhao, X., & Ge, S. (2026) · Multimedia Systems · doi:10.1007/s00530-026-02625-y
Sutantio, S. J., Yosephus, O. T., & Bandong, S. (2026) · IEEE International Conference on Instrumentation, Networks, Future Sustainability, Information Theory, Transportation, and Systems (INFINITYS) · doi:10.1109/infinitys69204.2026.11665967
Kurscheid, S., Reichelt, C., Geng, P., Tanz, L., Eder, D., & Daub, R. (2026) · Procedia CIRP · doi:10.1016/j.procir.2026.03.195
Think-Commit Decoupling for Accelerated and Reliable Tool Invocation in LLM Agents
Lu, X., Wang, J., He, Y., Wu, W., & Li, Y. (2026) · International Conference on Computer Information Science and Application Technology (CISAT) · doi:10.1109/cisat71030.2026.11682253
Self-citationShi, Y., Liu, Y., Du, G., Liu, H., & Feng, S. (2026) · IEEE Access · doi:10.1109/access.2026.3736562
Zeng, C., & Kang, S. (2026) · Asia Conference on Machine Learning and Computing (ACMLC) · doi:10.1109/acmlc70381.2026.11700470
Controlled knowledge updating in memory-enabled AI agents: beyond model editing and recall
Chavira, G., Bravo, E. J. G., García, G. E. R., Espinosa, J. A. A., & Nava, S. W. (2026) · Frontiers in Big Data · doi:10.3389/fdata.2026.1938137
A systematic mapping study of AI agent–Tool communication: protocols, security, and orchestration
UMER, H. S., Nawaz, N. A. H., Ishaq, K., & Ahmad, A. (2026) · Intelligent Systems with Applications · doi:10.1016/j.iswa.2026.200733
Jeong, Y., Cho, K.-Y., & Woo, J. H. (2026) · IFIP advances in information and communication technology · doi:10.1007/978-3-032-38614-4_36
Conflict-aware uncertainty alignment for tool-augmented document-centric multimodal intelligence
Muthusami, R., & Saritha, K. (2026) · Discover Artificial Intelligence · doi:10.1007/s44163-026-02409-3
Developing agentic AI systems for Earth observation data: A case study of the Living Wales data cubePreprint
Hountondji, O., Clewley, D., Nwokocha, E., Lucas, R., & Williams, H. T. P. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.7625215
Multi-level Retrieval-Augmented Generation for Agent Workflow Synthesis
Fan, H., & Cao, J. (2026) · Lecture notes in computer science · doi:10.1007/978-981-92-6910-5_32
EFSA-TD: Event-Level Financial Sentiment Analysis with Trigger Detection
Liu, S., Liu, H., Wang, X., Meng, F., Zhu, S., He, C., Luo, J., & Jia, Y. (2026) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn69854.2026.11730061
Structured Reinforcement Learning for Reliable Tool Invocation in Remote Sensing Agents
Yu, Z., Wei, Z., Peng, M., Wang, J., & Xu, W. (2026) · IEEE International Geoscience and Remote Sensing Symposium (IGARSS) · doi:10.1109/igarss60577.2026.11730704
Feng, B., Zhang, H., Zhang, C.-Y., Zhai, C., & Zhang, B. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.7674281
Grattafiori, A., Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Vaughan, A., et al. (2024) · arXiv · doi:10.48550/arxiv.2407.21783 · Cited by 505
Self-citationZhu, X., Chen, Y., Tian, H., Tao, C., Su, W., Yang, C., Huang, G., Li, B., Lu, L., Wang, X., et al. (2023) · arXiv · doi:10.48550/arxiv.2305.17144 · Cited by 33
Prompt Injection Attack to Tool Selection in LLM Agents
Shi, J., Yuan, Z., Tie, G., Zhou, P., Gong, N. Z., & Sun, L. (2025) · arXiv · doi:10.48550/arxiv.2504.19793 · Cited by 2
Metadata
- Authors
- Li, M., Zhao, Y., Yu, B., Song, F., Li, H., Yu, H., Li, Z., Huang, F., & Li, Y.
- Affiliations
- Alibaba Group (China) (CN)
- Hong Kong University of Science and Technology (HK)
- Peking University (CN)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2023
- Pages
- 3102-3116
- arXiv
- 2304.08244
- OpenAlex
- W4389518608
- Cited by
- 124
- References
- 27
- Citations by year
- 2026: 57 · 2025: 47 · 2024: 14 · 2023: 6
- FWCI
- 21.97
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- Rare & Special e-Zone (The Hong Kong University of Science and Technology) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- benchmark, error analysis
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
