StableToolBench: Towards Stable Large-Scale Benchmarking on Tool Learning of Large Language Models
Open accessAbstract
Large Language Models (LLMs) have witnessed remarkable advancements in recent years, prompting the exploration of tool learning, which integrates LLMs with external tools to address diverse real-world challenges. Assessing the capability of LLMs to utilise tools necessitates large-scale and stable benchmarks. However, previous works relied on either hand-crafted online tools with limited scale, or large-scale real online APIs suffering from instability of API status. To address this problem, we introduce StableToolBench, a benchmark evolving from ToolBench, proposing a virtual API server and stable evaluation system. The virtual API server contains a caching system and API simulators which are complementary to alleviate the change in API status. Meanwhile, the stable evaluation system designs solvable pass and win rates using GPT-4 as the automatic evaluator to eliminate the randomness during evaluation. Experimental results demonstrate the stability of StableToolBench, and further discuss the effectiveness of API simulators, the caching system, and the evaluator system.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
No references could be resolved yet.
Research works printed but not linked: 28
- Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language Models are Few-Shot Learners. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Pro- 9 11151
- cessing Systems 2020, NeurIPS 2020, December 6- 12, 2020, virtual.
- Tianle Cai, Xuezhi Wang, Tengyu Ma, Xinyun Chen, and Denny Zhou. 2024. Large Language Models as Tool Makers. In Proc. of The Twelfth Inter- national Conference on Learning Representations (ICLR 2024).
- Baian Chen, Chang Shu, Ehsan Shareghi, Nigel Col- lier, Karthik Narasimhan, and Shunyu Yao. 2023a. FireAct: Toward language agent fine-tuning. ArXiv preprint, abs/2310.05915. · arXiv
- Zehui Chen, Weihua Du, Wenwei Zhang, Kuikun Liu, Jiangning Liu, Miao Zheng, Jingming Zhuo, Songyang Zhang, Dahua Lin, Kai Chen, et al. 2023b. T-Eval: Evaluating the Tool Utilization Capability Step by Step. ArXiv preprint, abs/2312.14033. · arXiv
- Tanmay Gupta and Aniruddha Kembhavi. 2023. Vi- sual programming: Compositional visual reasoning without training. In In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog- nition (CVPR 2023), pages 14953-14962.
- Shibo Hao, Tianyang Liu, Zhen Wang, and Zhiting Hu. 2023. ToolkenGPT: Augmenting Frozen Language Models with Massive Tools via Tool Embeddings. ArXiv preprint, abs/2305.11554. · arXiv
- Cheng-Yu Hsieh, Si-An Chen, Chun-Liang Li, Yasuhisa Fujii, Alexander Ratner, Chen-Yu Lee, Ranjay Kr- ishna, and Tomas Pfister. 2023. Tool documenta- tion enables zero-shot tool-usage with large language models. ArXiv preprint, abs/2308.00675. · arXiv
- Yue Huang, Jiawen Shi, Yuan Li, Chenrui Fan, Siyuan Wu, Qihui Zhang, Yixin Liu, Pan Zhou, Yao Wan, Neil Zhenqiang Gong, et al. 2023. MetaTool bench- mark for large language models: Deciding whether to use tools and which to use. ArXiv preprint, abs/2310.03128. · arXiv
- Qiao Jin, Yifan Yang, Qingyu Chen, and Zhiyong Lu. 2024. GeneGPT: augmenting large language models with domain tools for improved access to biomedical information. Bioinformatics, 40(2):btae075. · DOI · Paper page
- Minghao Li, Feifan Song, Bowen Yu, Haiyang Yu, Zhoujun Li, Fei Huang, and Yongbin Li. 2023b. API-Bank: A Comprehensive Benchmark for Tool- Augmented LLMs. ArXiv preprint, abs/2304.08244. · arXiv
- Pan Lu, Baolin Peng, Hao Cheng, Michel Galley, Kai- Wei Chang, Ying Nian Wu, Song-Chun Zhu, and Jian- feng Gao. 2023. Chameleon: Plug-and-Play Com- positional Reasoning with Large Language Models. ArXiv preprint, abs/2304.09842. · arXiv
- Leland McInnes, John Healy, Nathaniel Saul, and Lukas Grossberger. 2018. UMAP: Uniform Manifold Ap- proximation and Projection. The Journal of Open Source Software, 3(29):861.
- Grégoire Mialon, Roberto Dessì, Maria Lomeli, Christo- foros Nalmpantis, Ram Pasunuru, Roberta Raileanu, Baptiste Rozière, Timo Schick, Jane Dwivedi-Yu, Asli Celikyilmaz, et al. 2023. Augmented Language Models: A Survey. ArXiv preprint, abs/2302.07842. · arXiv
- Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, Xu Jiang, Karl Cobbe, Tyna Eloundou, Gretchen Krueger, Kevin Button, Matthew Knight, Benjamin Chess, and John Schulman. 2022. We- bGPT: Browser-assisted question-answering with hu- man feedback. OpenAI. 2023. GPT-4 Technical Report.
- Shishir G. Patil, Tianjun Zhang, Xin Wang, and Joseph E. Gonzalez. 2023. Gorilla: Large Lan- guage Model Connected with Massive APIs. ArXiv preprint, abs/2305.15334. · arXiv
- Yujia Qin, Zihan Cai, Dian Jin, Lan Yan, Shihao Liang, Kunlun Zhu, Yankai Lin, Xu Han, Ning Ding, Huadong Wang, et al. 2023a. WebCPM: Interac- tive Web Search for Chinese Long-form Question Answering. ArXiv preprint, abs/2305.06849. · arXiv
- Yujia Qin, Shengding Hu, Yankai Lin, Weize Chen, Ning Ding, Ganqu Cui, Zheni Zeng, Yufei Huang, Chaojun Xiao, Chi Han, et al. 2023b. Tool learning with foundation models. ArXiv preprint, abs/2304.08354. · arXiv
- Nils Reimers and Iryna Gurevych. 2019. Sentence- BERT: Sentence embeddings using Siamese BERT- networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natu- ral Language Processing (EMNLP-IJCNLP), pages 3982-3992, Hong Kong, China. Association for Com- putational Linguistics. · DOI · Paper page
- Jingqing Ruan, Yihong Chen, Bin Zhang, Zhiwei Xu, Tianpeng Bao, Guoqing Du, Shiwei Shi, Hangyu Mao, Xingyu Zeng, and Rui Zhao. 2023. TPTU: Task planning and tool usage of large language model- based AI agents. ArXiv preprint, abs/2308.03427. · arXiv
- Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. 2023. Toolformer: Language Models Can Teach Themselves to Use Tools. ArXiv preprint, abs/2302.04761. 10 11152 · arXiv
- Yifan Song, Weimin Xiong, Dawei Zhu, Cheng Li, Ke Wang, Ye Tian, and Sujian Li. 2023. Rest- GPT: Connecting Large Language Models with Real-World Applications via RESTful APIs. ArXiv preprint, abs/2306.06624. · arXiv
- Alan M. Turing. 2009. Computing Machinery and In- telligence, pages 23-65. Springer Netherlands, Dor- drecht. · DOI
- Xingyao Wang, Zihan Wang, Jiateng Liu, Yangyi Chen, Lifan Yuan, Hao Peng, and Heng Ji. 2023. MINT: Evaluating LLMs in multi-turn interaction with tools and language feedback. ArXiv preprint, abs/2309.10691. · arXiv
- Linyao Yang, Hongyang Chen, Zhao Li, Xiao Ding, and Xindong Wu. 2023a. ChatGPT is not Enough: Enhancing Large Language Models with Knowledge Graphs for Fact-aware Language Modeling. ArXiv preprint, abs/2306.11489. · arXiv
- Shunyu Yao, Howard Chen, John Yang, and Karthik Narasimhan. 2022a. WebShop: Towards Scalable Real-World Web Interaction with Grounded Lan- guage Agents. In In Proceedings of the Advances in Neural Information Processing Systems (NeurIPS 2022), volume 35, pages 20744-20757.
- Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. 2022b. ReAct: Synergizing reasoning and acting in language models. volume abs/2210.03629. · arXiv
- Yuchen Zhuang, Yue Yu, Kuan Wang, Haotian Sun, and Chao Zhang. 2023. ToolQA: A Dataset for LLM Question Answering with External Tools. ArXiv preprint, abs/2306.13304. 11 11153 · arXiv
Cited by
Sorted by citations
When2Call: When (not) to Call Tools
Ross, H., Mahabaleshwarkar, A. S., & Suhara, Y. (2025) · Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) · doi:10.18653/v1/2025.naacl-long.174 · Cited by 4
Understanding and Optimizing Agentic Workflows via Shapley value
Yang, Y., Huang, B., Qi, S., Feng, C., Hu, H., Zhu, Y., Hu, J., Zhao, H., He, Z., Liu, X., et al. (2025) · arXiv · doi:10.48550/arxiv.2502.00510 · Cited by 2 · see citing works
Lu, J., Holleis, T., Zhang, Y., Aumayer, B., Nan, F., Bai, F., Ma, S., Ma, S., Li, M., Yin, G., et al. (2024) · arXiv · doi:10.48550/arxiv.2408.04682 · Cited by 2 · see citing works
Measuring AI Ability to Complete Long Software Tasks
Kwa, T., West, B., Becker, J., Deng, A., Garcia, K., Hasin, M., Jawhar, S., Kinniment, M., Rush, N., Von Arx, S., et al. (2025) · Advances in Neural Information Processing Systems 38 · doi:10.52202/085713-3086 · Cited by 1 · see citing works
“MLAgentBench [14] and MLEBench [15] focus specifically on machine learning research tasks, while ToolBench [33; 10] assesses tool use capabilities.”
Preble: Efficient Distributed Prompt Scheduling for LLM Serving
Srivatsa, V., He, Z., Abhyankar, R., Li, D., & Zhang, Y. (2024) · arXiv · doi:10.48550/arxiv.2407.00023 · Cited by 1 · see citing works
“We evaluate the Toolbench [11] dataset, which consists of more than 210k queries that call over 16k unique tools.”
MethodVehicleMemBench: An Executable Benchmark for Multi-User Long-Term Memory in In-Vehicle Agents
Chen, Y., Xu, Y., Ding, X., Fang, X., Liu, S., Lin, L., Zhang, Q., Li, Y., Liu, Q., & Xu, T. (2026) · arXiv · doi:10.48550/arxiv.2603.23840
“On the other hand, tool use benchmarks [Chen et al., 2024, Guo et al., 2025, Yang et al., 2025b] typically emphasize short-horizon instruction following, with insufficient consideration of long-term, evolving user preferences.”
HCAST: Human-Calibrated Autonomy Software Tasks
Rein, D., Becker, J., Deng, A., Nix, S., Canal, C., O'Connel, D., Arnott, P., Bloom, R., Broadley, T., Garcia, K., et al. (2025) · arXiv · doi:10.48550/arxiv.2503.17354
“ToolSandbox (Lu et al., 2024), ToolBench (Guo et al., 2024), and the Berkeley Function Calling Leaderboard (Yan et al., 2024) evaluate the ability of AI agents to interface with provided tools or APIs.”
From Word to World: Can Large Language Models be Implicit Text-based World Models?
Li, Y., Wang, H., Qiu, J., Yin, Z., Zhang, D., Qian, C., Li, Z., Ma, X., Chen, G., & Ji, H. (2026) · Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2026.acl-long.366
Self-citationEvaluating and Guarding Citation Faithfulness in Agentic Scientific Synthesis
Goo, T., Kim, J., Han, K., Jo, G., Kim, J.-S., & Kim, T.-H. (2026) · arXiv · doi:10.48550/arxiv.2607.20527
Self-Challenging Language Model Agents
Zhou, Y., Levine, S., Weston, J., Li, X., & Sukhbaatar, S. (2025) · arXiv · doi:10.48550/arxiv.2506.01716 · Cited by 1
- Authors
- Guo, Z., Cheng, S., Wang, H., Liang, S., Qin, Y., Li, P., Liu, Z., Sun, M., & Liu, Y.
- Affiliations
- Tsinghua University (CN)
- Google (United States) (US)
- University of Hong Kong (HK)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2024
- Pages
- 11143-11156
- arXiv
- 2403.07714
- OpenAlex
- W4402670152
- Cited by
- 22
- References
- 0
- Citations by year
- 2026: 10 · 2025: 11 · 2024: 1
- FWCI
- 6.35
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Findings of the Association for Computational Linguistics: ACL (Findings of ACL) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Natural Science Foundation of China (62276152, 61925601)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
- Keywords
- tool learning, large language models, benchmarking, automated assessment, GPT-4
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
