TaskBench: Benchmarking Large Language Models for Task Automation
Open accessAbstract
In recent years, the remarkable progress of large language models (LLMs) has sparked interest in task automation, which involves decomposing complex tasks described by user instructions into sub-tasks and invoking external tools to execute them, playing a central role in autonomous agents. However, there is a lack of systematic and standardized benchmarks to promote the development of LLMs in task automation. To address this, we introduce TaskBench, a comprehensive framework to evaluate the capability of LLMs in task automation. Specifically, task automation can be divided into three critical stages: task decomposition, tool selection, and parameter prediction. To tackle the complexities inherent in these stages, we introduce the concept of Tool Graph to represent decomposed tasks and adopt a back-instruct method to generate high-quality user instructions. We propose TaskEval, a multi-faceted evaluation methodology that assesses LLM performance across these three stages. Our approach combines automated construction with rigorous human verification, ensuring high consistency with human evaluation. Experimental results demonstrate that TaskBench effectively reflects the capabilities of various LLMs in task automation. It provides insights into model performance across different task complexities and domains, pushing the boundaries of what current models can achieve. TaskBench offers a scalable, adaptable, and reliable benchmark for advancing LLM-based autonomous agents.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Datasets such as AAAR-1.0, ScienceAgentBench, and TaskBench provide structured, expert-labeled benchmarks for assessing research reasoning, scientific workflows, and multi-tool planning.
“For instance, datasets such as AAAR-1.0 [61], ScienceAgentBench [11], and TaskBench [83] provide structured, expert-labeled benchmarks for assessing research reasoning, scientific workflows, and multi-tool planning.”
From Evaluation and Benchmarking of LLM Agents: A Survey (2025) · cites this paper as [83]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
The publisher hasn’t deposited this paper’s reference list.
Research works printed but not linked: 5
- Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In NeurIPS, pages 1-25, 2020.
- Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F. Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. In NeurIPS, volume 35, pages 27730-27744, 2022.
- Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In ICLR, 2021.
- Yue Huang, Jiawen Shi, Yuan Li, Chenrui Fan, Siyuan Wu, Qihui Zhang, Yixin Liu, Pan Zhou, Yao Wan, Neil Zhenqiang Gong, and Lichao Sun. Metatool benchmark: Deciding whether to use tools and which to use. arXiv preprint arXiv: 2310.03128, 2023. · arXiv
- Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language models with self-generated in- structions. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 13484-13508, Toronto, Canada, July 2023. Associ- ation for Computational Linguistics.
Cited by
Sorted by citations
A Survey of GPT-3 Family Large Language Models Including ChatGPT and GPT-4
Kalyan, K. S. (2024) · Natural Language Processing Journal · doi:10.1016/j.nlp.2023.100048 · Cited by 335 · see citing works
Tool learning with large language models: a survey
Qu, C., Dai, S., Wei, X., Cai, H., Wang, S., Yin, D., Xu, J., & Wen, J.-R. (2025) · Frontiers of Computer Science · doi:10.1007/s11704-024-40678-2 · Cited by 148 · see citing works
Evaluation and Benchmarking of LLM Agents: A Survey
Mohammadi, M., Li, Y., Lo, J., & Yip, W. (2025) · Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.2 · doi:10.1145/3711896.3736570 · Cited by 74 · see citing works
“The parameter name F1 score [83] measures the agent’s ability to identify the parameter names required for a given function correctly and then correctly assign values to them.”
InfluentialAdapting LLMs for Efficient Context Processing through Soft Prompt Compression
Wang, C., Yang, Y., Li, R., Sun, D., Cai, R., Zhang, Y., & Fu, C. (2024) · International Conference on Modeling, Natural Language Processing and Machine Learning (CMNM) · doi:10.1145/3677779.3677794 · Cited by 47 · see citing works
A Framework for Integrated Digital Forensic Investigation Employing AutoGen AI Agents
Wickramasekara, A., & Scanlon, M. (2024) · International Symposium on Digital Forensics and Security (ISDFS) · doi:10.1109/isdfs60797.2024.10527235 · Cited by 21 · see citing works
Chen, Z., Hao, J., Sun, H., Zhao, L., Li, J., Qian, Q., Peng, Q., Wang, X., Cong, S., Shen, L., et al. (2025) · Information Processing & Management · doi:10.1016/j.ipm.2025.104278 · Cited by 17 · see citing works
DAWN: Designing Distributed Agents in a Worldwide Network
Aminiranjbar, Z., Tang, J., Wang, Q., Pant, S., & Viswanathan, M. (2025) · IEEE Access · doi:10.1109/access.2025.3588425 · Cited by 8 · see citing works
Kapoor, S., Stroebl, B., Siegel, Z. S., Nadgir, N., & Narayanan, A. (2024) · arXiv · doi:10.48550/arxiv.2407.01502 · Cited by 13 · see citing works
“TaskBench[42] Task automation Measurestheaccuracyofagentsonasetoftaskswithinthetaskautomationdomain.”
Action Engine: Automatic Workflow Generation in FaaS
Esashi, A., Lertpongrujikorn, P., Kato, S., & Salehi, M. A. (2025) · Future Generation Computer Systems · doi:10.1016/j.future.2025.107947 · Cited by 6 · see citing works
Chen, M., Fan, S., Cao, G., Liu, Y., & Liu, H. (2025) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2025.3598625 · Cited by 6 · see citing works
Go, H., & Park, S. (2025) · Scientific Reports · doi:10.1038/s41598-025-06469-w · Cited by 4 · see citing works
Gabriel, A. G., Ahmad, A. A., & Jeyakumar, S. K. (2024) · arXiv · doi:10.48550/arxiv.2410.22457 · Cited by 4 · see citing works
Sun, Y., Wang, X., Xu, H., & Wang, Z. (2026) · Lecture notes in computer science · doi:10.1007/978-981-95-5012-8_5 · Cited by 3 · see citing works
AppWorld: A Controllable World of Apps and People for Benchmarking Interactive Coding Agents
Trivedi, H., Khot, T., Hartmann, M., Manku, R., Dong, V., Li, E., Gupta, S., Sabharwal, A., & Balasubramanian, N. (2024) · arXiv · doi:10.48550/arxiv.2407.18901 · Cited by 2 · see citing works
Dadu‐E : Rethinking the Role of Large Language Model in Robotic Computing Pipelines
Sun, W., Hou, S., Wang, Z., Bo, Y., Liu, S., Yang, X., Liang, S., Gan, Y., & Han, Y. (2025) · Journal of Field Robotics · doi:10.1002/rob.70120 · Cited by 3 · see citing works
The Illusion of Diminishing Returns: Measuring Long Horizon Execution in LLMs
Sinha, A., Arun, A., Goel, S., Staab, S., & Geiping, J. (2025) · arXiv · doi:10.48550/arxiv.2509.09677 · Cited by 1 · see citing works
“Recently, multiple real-world long-horizon agentic benchmarks have been proposed (Backlund & Petersson, 2025; Xie et al., 2024; Shen et al., 2025), where prior work has studied planning failures (Chen et al., 2024b).”
Towards Efficient Compound Large Language Model System Serving in the Wild
Zhu, Y., Zhu, B., Chen, C., & Fan, X. (2024) · International Workshop on Quality of Service · doi:10.1109/iwqos61813.2024.10682929 · Cited by 1 · see citing works
Harnessing LLMs for VQA: A Prompted Benchmark with Animate/Inanimate Keywords
Lee, C., Lee, H., Kim, M., Kim, H. W., Jang, H., & Park, C. (2024) · International Conference on Information and Communication Technology Convergence (ICTC) · doi:10.1109/ictc62082.2024.10826596 · Cited by 1 · see citing works
AJ-Bench: Benchmarking Agent-as-a-Judge for Environment-Aware Evaluation
Shi, W., Wang, Y., Zhao, Y., Chen, Y., Feng, F., Hao, X., Su, X., Gu, Q., Su, H., Cai, X., et al. (2026) · arXiv · doi:10.48550/arxiv.2604.18240
“…(Trivedi et al., 2024)), tool-using agents that stress multi-tool planning over large APIs (e.g., ToolBench (Huang et al., 2023), API-Bank (Li et al., 2023)), as well as domain-specific, robustness, and safety settings (e.g., ScienceAgent-Bench (Chen et al., 2024), TaskBench (Shen et al., 2024)).”
Chung, J., Lee, M., & Kim, J. (2025) · International Conference on Information and Communication Technology Convergence (ICTC) · doi:10.1109/ictc66702.2025.11387844
ReGRPO: Reflection-Augmented Policy Optimization for Tool-Using Agents
Zhang, B., & Shou, M. Z. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-37174-4_7
MGFS: Structure-Aware Multi-granular Function Selection for LLM-Based Agents
Zhang, Y., Li, C., Ma, J., Wang, Q., Yang, W., & Zhao, Z. (2026) · Lecture notes in computer science · doi:10.1007/978-981-92-3403-5_11
Herding CATs: ALARA for Agent Harness Engineering in Portable Composable Multi-Agent Teams
Agostino, C. J., & D'Souza, N. (2026) · International Conference on Human-AI Interaction and Experience Design (HAXD) · doi:10.1109/haxd70072.2026.11621027
Local LLMs as cooperative agents for low-cost surgical navigation support
Barr, C., Galvin, C., Azimi, A., Frisken, S., Pieper, S., Gabor Fichtinger, A., Golby, A. J., & Mousavi, P. (2026) · International Journal of Computer Assisted Radiology and Surgery · doi:10.1007/s11548-026-03772-3
TopoMLP++: Toward Strong and Scalable Lane Topology Reasoning in Autonomous Driving
Wu, D., Han, W., Xu, C., & Shen, J. (2026) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2026.3721163
The Illusion of Success: Procedure-Aware Evaluation is Critical for LLM Agents
Cao, H., Driouich, I., & Thomas, E. M. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-37682-4_29 · Cited by 5 · see citing works
Integration of Process Mining and Large Language Models—Literature Review
Halaška, M., Šperka, R., Dolák, R., & Barcak, T. (2026) · Smart innovation, systems and technologies · doi:10.1007/978-3-032-11456-3_16
Song, Y., & Lyu, H. (2026) · Robotics and Autonomous Systems · doi:10.1016/j.robot.2026.105815
Digital Twins and Predictive Intelligence for Smart Tourism and Travel Operations
Sharma, A., Kataria, P., Bisla, N., Kumari, S., & Pham, U. M. T. (2026) · Advances in computational intelligence and robotics book series · doi:10.4018/979-8-2600-4962-4.ch001
SUPER: Evaluating Agents on Setting Up and Executing Tasks from Research Repositories
Bogin, B., Yang, K., Gupta, S., Richardson, K., Bransom, E., Clark, P., Sabharwal, A., & Khot, T. (2024) · arXiv · doi:10.48550/arxiv.2409.07440
- Authors
- Shen, Y., Song, K., Tan, X., Zhang, W., Ren, K., Yuan, S., Lu, W., Li, D., & Zhuang, Y.
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2023
- Pages
- 4540-4574
- ISSN
- 1049-5258 (ISSN-L)
- arXiv
- 2311.18760
- OpenAlex
- W4415798810
- Cited by
- 24
- References
- 0
- Citations by year
- 2026: 10 · 2025: 9 · 2024: 4 · 2023: 1
- FWCI
- 6.79
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- neural information processing systems (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Artificial Intelligence Applications
- Keywords
- large language models, benchmarking, task automation, AI agents
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
