When2Call: When (not) to Call Tools
Open accessAbstract
Hayley Ross, Ameya Sunil Mahabaleshwarkar, Yoshi Suhara. Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers). 2025.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Skill discovery is a higher-level matching problem that must consider task complexity, environmental assumptions, operational constraints, and risk conditions, not just topic similarity.
“This makes skill discovery a higher-level matching problem. It must consider not only topic similarity, but also task complexity, environmental assumptions, operational constraints, and risk conditions.”
From Externalization in LLM Agents: A Unified Review of Memory, Skills, Protocols and Harness Engineering (2026) · cites this paper as Ross et al., 2025
Section 2.4 shows that a question and tool can share a semantic domain yet still be mismatched, and Section 2.2 discusses checking tool capabilities and missing required parameters. This supports the point that topic similarity alone is insufficient. The paper does not discuss skill discovery as such or establish the claimed need to consider task complexity, environmental assumptions, operational constraints, and risk conditions. (LLM, conf=0.84)
Experimental — accuracy not yet measuredSkill effectiveness depends heavily on alignment between task intent, skill description, and invocation decision; mismatches can cause a model to follow literal wording while missing the real objective.
“As a result, a model may follow the literal wording of a skill while still missing the real objective of the task.”
From Externalization in LLM Agents: A Unified Review of Memory, Skills, Protocols and Harness Engineering (2026) · cites this paper as Ross et al., 2025
Sections 2.4 and 3.2 show that subtle mismatches between a user's question and a tool can lead models to make unwanted tool calls rather than address what the question requires. The paper does not establish the more specific claim that a model follows a skill's literal wording while missing the task's real objective; it studies tool selection, not skills or literal adherence to their wording. (LLM, conf=0.82)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Gorilla: Large Language Model Connected with Massive APIs
Patil, S. G., Zhang, T., Wang, X., & Gonzalez, J. E. (2023) · arXiv · doi:10.48550/arxiv.2305.15334 · Cited by 226 · see citing works
API-Bank: A Comprehensive Benchmark for Tool-Augmented LLMs
Li, M., Zhao, Y., Yu, B., Song, F., Li, H., Yu, H., Li, Z., Huang, F., & Li, Y. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.187 · Cited by 124 · see citing works
Large Language Models Sensitivity to The Order of Options in Multiple-Choice Questions
Pezeshkpour, P., & Hruschka, E. (2024) · Findings of the Association for Computational Linguistics: NAACL 2024 · doi:10.18653/v1/2024.findings-naacl.130 · Cited by 85 · see citing works
Qin, Y., Liang, S., Ye, Y., Zhu, K., Yan, L., Lu, Y., Lin, Y., Cong, X., Tang, X., Qian, B., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2307.16789 · Cited by 71 · see citing works
xLAM: A Family of Large Action Models to Empower AI Agent Systems
Zhang, J., Lan, T., Zhu, M. H., Liu, Z., Hoang, T. Q., Kokane, S., Yao, W., Tan, J., Prabhakar, A., Chen, H., et al. (2025) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2025.naacl-long.578 · Cited by 58 · see citing works
Ye, J., Li, G., Gao, S., et al. (2024) · arXiv.org · doi:10.48550/arxiv.2401.00741 · Cited by 44 · see citing works
StableToolBench: Towards Stable Large-Scale Benchmarking on Tool Learning of Large Language Models
Guo, Z., Cheng, S., Wang, H., Liang, S., Qin, Y., Li, P., Liu, Z., Sun, M., & Liu, Y. (2024) · arXiv · doi:10.48550/arxiv.2403.07714 · Cited by 22 · see citing works
ToolAlpaca: Generalized Tool Learning for Language Models with 3000 Simulated Cases
Tang, Q., Deng, Z., Lin, H., Han, X., Liang, Q., Cao, B., & Sun, L. (2023) · arXiv · doi:10.48550/arxiv.2306.05301 · Cited by 16 · see citing works
APIGen: Automated Pipeline for Generating Verifiable and Diverse Function-Calling Datasets
Liu, Z., Hoang, T., Zhang, J., Zhu, M., Lan, T., Kokane, S., Tan, J., Yao, W., Liu, Z., Feng, Y., et al. (2024) · arXiv · doi:10.48550/arxiv.2406.18518 · Cited by 13 · see citing works
On the Tool Manipulation Capability of Open-source Large Language Models
Xu, Q., Hong, F., Li, B., Hu, C., Chen, Z., & Zhang, J. (2023) · arXiv · doi:10.48550/arxiv.2305.16504 · Cited by 11 · see citing works
The Art of Refusal: A Survey of Abstention in Large Language Models
Wen, B.-B., Yao, J.-H., Feng, S.-B., et al. (2024) · arXiv.org · doi:10.48550/arxiv.2407.18418 · Cited by 9
$τ$-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains
Yao, S., Shinn, N., Razavi, P., & Narasimhan, K. (2024) · arXiv · doi:10.48550/arxiv.2406.12045 · Cited by 9 · see citing works
MetaTool Benchmark for Large Language Models: Deciding Whether to Use Tools and Which to Use
Huang, Y., Shi, J., Li, Y., Fan, C., Wu, S., Zhang, Q., Liu, Y., Zhou, P., Wan, Y., Gong, N. Z., et al. (2023) · arXiv · doi:10.48550/arxiv.2310.03128 · Cited by 7 · see citing works
Lu, J., Holleis, T., Zhang, Y., Aumayer, B., Nan, F., Bai, F., Ma, S., Ma, S., Li, M., Yin, G., et al. (2024) · arXiv · doi:10.48550/arxiv.2408.04682 · Cited by 2 · see citing works
Tool Learning with Large Language Models: A Survey
Qu, C., Dai, S., Wei, X., et al. (2024) · arXiv · doi:10.48550/arxiv.2405.17935
Flagged first, then by citations
Research works printed but not linked: 13
- Ibrahim Abdelaziz, Kinjal Basu, Mayank Agarwal, Sadhana Kumaravel, Matthew Stallone, Rameswar Panda, Yara Rizk, G. P. Bhargav, Maxwell Crouse, Chulaka Gunasekara, Shajith Ikbal, Sachin Joshi, Hima Karanam, Vineet Kumar, Asim Munawar, Sumit Neelam, Dinesh Raghu, Udit Sharma, Adriana Meza Soria, Dheeraj Sreedhar, Praveen Venkateswaran, Merve Unuvar, David Cox, Salim Roukos, Luis Lastras, and Pavan Kapanipathi. 2024. Granite-Function Calling Model: Introducing Func- tion Calling Abilities via Multi-task Learning of Granular Tasks. arXiv preprint. ArXiv:2407.00121 [cs] version: 1. · DOI
- Kinjal Basu, Ibrahim Abdelaziz, Subhajit Chaudhury, Soham Dan, Maxwell Crouse, Asim Munawar, Ver- non Austel, Sadhana Kumaravel, Vinod Muthusamy, Pavan Kapanipathi, and Luis Lastras. 2024. API- BLEND: A comprehensive corpora for training and benchmarking API LLMs. In Proceedings of the 62nd Annual Meeting of the Association for Compu- tational Linguistics (Volume 1: Long Papers), pages 12859-12870, Bangkok, Thailand. Association for Computational Linguistics. · DOI
- Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, and et al. 2024. The llama 3 herd of models. Preprint, arXiv:2407.21783. · arXiv
- Vipul Gupta, David Pantoja, Candace Ross, Adina Williams, and Megan Ung. 2024. Changing an- swer order can decrease mmlu accuracy. Preprint, arXiv:2406.19470. · arXiv
- Nvidia, :, Bo Adler, Niket Agarwal, Ashwath Aithal, Dong H. Anh, Pallab Bhattacharya, Annika Brun- dyn, Jared Casper, Bryan Catanzaro, Sharon Clay, Jonathan Cohen, Sirshak Das, Ayush Dattagupta, Olivier Delalleau, Leon Derczynski, Yi Dong, Daniel Egert, Ellie Evans, Aleksander Ficek, Denys Frid- man, Shaona Ghosh, Boris Ginsburg, Igor Gitman, Tomasz Grzegorzek, Robert Hero, Jining Huang, Vibhu Jawa, Joseph Jennings, Aastha Jhunjhunwala, John Kamalu, Sadaf Khan, Oleksii Kuchaiev, Patrick LeGresley, Hui Li, Jiwei Liu, Zihan Liu, Eileen Long, Ameya Sunil Mahabaleshwarkar, Somshubra Ma- jumdar, James Maki, Miguel Martinez, Maer Ro- drigues de Melo, Ivan Moshkov, Deepak Narayanan, Sean Narenthiran, Jesus Navarro, Phong Nguyen, Osvald Nitski, Vahid Noroozi, Guruprasad Nutheti, Christopher Parisien, Jupinder Parmar, Mostofa Pat- wary, Krzysztof Pawelec, Wei Ping, Shrimai Prabhu- moye, Rajarshi Roy, Trisha Saar, Vasanth Rao Naik Sabavat, Sanjeev Satheesh, Jane Polak Scowcroft, Jason Sewall, Pavel Shamis, Gerald Shen, Moham- mad Shoeybi, Dave Sizer, Misha Smelyanskiy, Fe- lipe Soares, Makesh Narsimhan Sreedhar, Dan Su, Sandeep Subramanian, Shengyang Sun, Shubham Toshniwal, Hao Wang, Zhilin Wang, Jiaxuan You, Ji- aqi Zeng, Jimmy Zhang, Jing Zhang, Vivienne Zhang, Yian Zhang, and Chen Zhu. 2024. Nemotron-4 340b technical report. Preprint, arXiv:2406.11704. · arXiv
- Michael C Rodriguez. 2005. Three options are optimal for multiple-choice items: A meta-analysis of 80 years of research. Educational measurement: issues and practice, 24(2):3-13.
- Gerald Shen, Zhilin Wang, Olivier Delalleau, Jiaqi Zeng, Yi Dong, Daniel Egert, Shengyang Sun, Jimmy Zhang, Sahil Jain, Ali Taghibakhshi, Markel Sanz Ausin, Ashwath Aithal, and Oleksii Kuchaiev. 2024. NeMo-Aligner: Scalable toolkit for efficient model alignment. Preprint, arXiv:2405.01481. · arXiv
- Sharath Turuvekere Sreenivas, Saurav Muralidharan, Raviraj Joshi, Marcin Chochowski, Ameya Sunil Ma- habaleshwarkar, Gerald Shen, Jiaqi Zeng, Zijia Chen, Yoshi Suhara, Shizhe Diao, Chenhan Yu, Wei-Chun Chen, Hayley Ross, Oluwatobi Olabiyi, Ashwath Aithal, Oleksii Kuchaiev, Daniel Korzekwa, Pavlo Molchanov, Mostofa Patwary, Mohammad Shoeybi, Jan Kautz, and Bryan Catanzaro. 2024. LLM pruning and distillation in practice: The Minitron approach. Preprint, arXiv:2408.11796. · arXiv
- Rui Yang, Lin Song, Yanwei Li, Sijie Zhao, Yixiao Ge, Xiu Li, and Ying Shan. 2023. GPT4Tools: Teach- ing Large Language Model to Use Tools via Self- instruction. Advances in Neural Information Process- ing Systems, 36:71995-72007.
- Yuxiang Zhang, Jing Chen, Junjie Wang, Yaxin Liu, Cheng Yang, Chufan Shi, Xinyu Zhu, Zihao Lin, Hanwen Wan, Yujiu Yang, Tetsuya Sakai, Tian Feng, and Hayato Yamana. 2024b. ToolBeHonest: A Multi- level Hallucination Diagnostic Benchmark for Tool- Augmented Large Language Models. arXiv preprint. · DOI
- Chujie Zheng, Hao Zhou, Fandong Meng, Jie Zhou, and Minlie Huang. 2024a. Large language models are not robust multiple choice selectors. In The Twelfth International Conference on Learning Representa- tions.
- Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2024b. Judging llm-as-a-judge with mt-bench and chatbot arena. In Proceedings of the 37th International Conference on Neural Information Processing Systems, NeurIPS '23, Red Hook, NY, USA.
- Yuchen Zhuang, Yue Yu, Kuan Wang, Haotian Sun, and Chao Zhang. 2023. ToolQA: A Dataset for LLM Question Answering with External Tools. Advances in Neural Information Processing Systems, 36:50117- 50143.
Cited by
Sorted by citations
Externalization in LLM Agents: A Unified Review of Memory, Skills, Protocols and Harness Engineering
Zhou, C., Chai, H., Chen, W., Guo, Z., Shan, R., Song, Y., Xu, T., Yang, Y., Yu, A., Zhang, W., et al. (2026) · arXiv · doi:10.48550/arxiv.2604.08224 · Cited by 1 · see citing works
RRTL: Red Teaming Reasoning Large Language Models in Tool Learning
Liu, Y., Cui, Y., & Zhang, H. (2025) · IEEE Internet of Things Journal · doi:10.1109/jiot.2025.3642164 · Cited by 2 · see citing works
Fedorov, A. M., Datyev, I. O., Илясов, М. О., Вишняков, И. Г., Bazegskiy, M. O., Figurkin, D. S., & Lyubimova, C. D. (2025) · Transaction Kola Science Centre · doi:10.37614/2949-1215.2025.16.3.001 · Cited by 2 · see citing works
Edge-Deployed Conversational LLM Agents for Semantic Service Matching
Scheffler, J., Dartmann, G., Koß, A., Multhaupt, S., & Fazlic, L. B. (2026) · IEEE International Conference on Semantic Computing (ICSC) · doi:10.1109/icsc67292.2026.00074
(2026) · Ìnformatika, kulʹtura, tehnìka. · doi:10.15276/ict.03.2026.18
Think-Commit Decoupling for Accelerated and Reliable Tool Invocation in LLM Agents
Lu, X., Wang, J., He, Y., Wu, W., & Li, Y. (2026) · International Conference on Computer Information Science and Application Technology (CISAT) · doi:10.1109/cisat71030.2026.11682253
- Authors
- Ross, H., Mahabaleshwarkar, A. S., & Suhara, Y.
- Affiliations
- Harvard University (US)
- Nvidia (United Kingdom) (GB)
- ORCID
- Ross, Hayley
- Venue
- Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)
- Published
- 2025
- Pages
- 3391-3409
- OpenAlex
- W4411120459
- Cited by
- 4
- References
- 1
- Citations by year
- 2026: 2 · 2025: 2
- FWCI
- 3.81
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) (publishedVersion, cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- tool calling, large language models
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
