FlowBench: Revisiting and Benchmarking Workflow-Guided Planning for LLM-based Agents
Open accessAbstract
LLM-based agents have emerged as promising tools, which are crafted to fulfill complex tasks by iterative planning and action. However, these agents are susceptible to undesired planning hallucinations when lacking specific knowledge for expertise-intensive tasks. To address this, preliminary attempts are made to enhance planning reliability by incorporating external workflow-related knowledge. Despite the promise, such infused knowledge is mostly disorganized and diverse in formats, lacking rigorous formalization and comprehensive comparisons. Motivated by this, we formalize different formats of workflow knowledge and present FlowBench, the first benchmark for workflow-guided planning. FlowBench covers 51 different scenarios from 6 domains, with knowledge presented in diverse formats. To assess different LLMs on FlowBench, we design a multi-tiered evaluation framework. We evaluate the efficacy of workflow knowledge across multiple formats, and the results indicate that current LLM agents need considerable improvements for satisfactory planning. We hope that our challenging benchmark can pave the way for future agent planning research.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Existing benchmarks for conversational AI agents simulate single-control environments, where only the AI agent can use tools to interact with the world, while the user remains a passive information provider.
“Existing benchmarks for conversational AI agents simulate single-control environments, where only the AI agent can use tools to interact with the world, while the user remains a passive information provider.”
From $τ^2$-Bench: Evaluating Conversational Agents in a Dual-Control Environment (2025) · cites this paper as [23]
Section 3.1 (page unavailable) says most existing agent studies give agents complete task descriptions initially, rather than allowing users to refine requirements through ongoing conversation. Section 4.3.2 says FlowBench’s multi-turn setting goes beyond the traditional agent paradigm. Neither passage establishes that existing conversational-agent benchmarks are single-control environments in which only agents can use tools and users are passive information providers. (LLM, conf=0.84)
Experimental — accuracy not yet measuredDatasets such as FlowBench, ToolBench, and API-Bank focus on tool use and function-calling across large API repositories.
“Others, such as FlowBench [96], ToolBench [38], and API-Bank [47], focus on tool use and function-calling across large API repositories.”
From Evaluation and Benchmarking of LLM Agents: A Survey (2025) · cites this paper as [96]
Section 4.2 says FlowBench includes tool-invocation information in GPT-4 function-calling format, so the tool-use and function-calling part applies. The Introduction identifies its focus as workflow-guided agent planning across 6 domains and 51 scenarios; our copy does not describe it as a benchmark across large API repositories. The FlowBench citation is assessed here, not the separate citations to ToolBench and API-Bank. (LLM, conf=0.88)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
LLaMA: Open and Efficient Foundation Language Models
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. (2023) · arXiv · doi:10.48550/arxiv.2302.13971 · Cited by 4,602 · see citing works
Retrieval-Augmented Generation for Large Language Models: A Survey
Gao, Y., Xiong, Y., Gao, X., Jia, K., Pan, J., Bi, Y., Dai, Y., Sun, J., Wang, M., & Wang, H. (2023) · arXiv · doi:10.48550/arxiv.2312.10997 · Cited by 876 · see citing works
Self-citationAutoAgents: A Framework for Automatic Agent Generation
Chen, G., Dong, S., Shu, Y., Zhang, G., Sesay, J., Karlsson, B. F., Fu, J., & Shi, Y.-M. (2023) · International Joint Conference on Artificial Intelligence · doi:10.48550/arxiv.2309.17288 · Cited by 324
Gorilla: Large Language Model Connected with Massive APIs
Patil, S. G., Zhang, T., Wang, X., & Gonzalez, J. E. (2023) · arXiv · doi:10.48550/arxiv.2305.15334 · Cited by 226 · see citing works
MetaGPT: Meta Programming for A Multi-Agent Collaborative Framework
Hong, S., Zhuge, M., Chen, J., Zheng, X., Cheng, Y., Zhang, C., Wang, J., Wang, Z., Yau, S. K. S., Lin, Z., et al. (2023) · arXiv · doi:10.48550/arxiv.2308.00352 · Cited by 214 · see citing works
WikiHow: A Large Scale Text Summarization Dataset
Koupaee, M., & Wang, W. Y. (2018) · arXiv · doi:10.48550/arxiv.1810.09305 · Cited by 166 · see citing works
KG-Agent: An Efficient Autonomous Agent Framework for Complex Reasoning over Knowledge GraphPreprint
Jiang, J.-H., Zhou, K., Zhao, W., et al. (2024) · Annual Meeting of the Association for Computational Linguistics · doi:10.48550/arxiv.2402.11163 · Cited by 151 · see citing works
Yang, K., Liu, J., Wu, J., Yang, C., Fung, Y. R., Li, S., Huang, Z.-X., Cao, X., Wang, X., Wang, Y.-Q., et al. (2024) · arXiv.org · doi:10.48550/arxiv.2401.00812 · Cited by 133 · see citing works
Generate-on-Graph: Treat LLM as both Agent and KG for Incomplete Knowledge Graph Question AnsweringPreprint
Xu, Y., He, S.-Z., Chen, J.-B., et al. (2024) · Conference on Empirical Methods in Natural Language Processing · doi:10.48550/arxiv.2404.14741 · Cited by 114 · see citing works
ChatDev: Communicative Agents for Software Development
Qian, C., Liu, W., Liu, H., Chen, N., Dang, Y., Li, J., Yang, C., Chen, W., Su, Y., Cong, X., et al. (2023) · arXiv · doi:10.48550/arxiv.2307.07924 · Cited by 81 · see citing works
AgentBench: Evaluating LLMs as Agents
Liu, X., Yu, H., Zhang, H., Xu, Y., Lei, X., Lai, H., Gu, Y., Ding, H., Men, K., Yang, K., et al. (2023) · arXiv · doi:10.48550/arxiv.2308.03688 · Cited by 79 · see citing works
Qin, Y., Liang, S., Ye, Y., Zhu, K., Yan, L., Lu, Y., Lin, Y., Cong, X., Tang, X., Qian, B., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2307.16789 · Cited by 71 · see citing works
REPLUG: Retrieval-augmented black-box language models
Shi, W., Min, S., Yasunaga, M., Seo, M., James, R., Lewis, M., Zettlemoyer, L., & Yih, W.-T. (2023) · arXiv · doi:10.48550/arxiv.2301.12652 · Cited by 65 · see citing works
STAR: A Schema-Guided Dialog Dataset for Transfer LearningPreprint
Mosig, J. E. M., Mehri, S., & Kober, T. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.11853 · Cited by 36 · see citing works
AgentBoard: An Analytical Evaluation Board of Multi-turn LLM Agents
Ma, C., Zhang, J., Zhu, Z., Yang, C., Yang, Y., Jin, Y., Lan, Z., Kong, L., & He, J. (2024) · arXiv · doi:10.48550/arxiv.2401.13178 · Cited by 26 · see citing works
TravelPlanner: A Benchmark for Real-World Planning with Language Agents
Xie, J., Zhang, K., Chen, J., Zhu, T., Lou, R., Tian, Y., Xiao, Y., & Su, Y. (2024) · arXiv · doi:10.48550/arxiv.2402.01622 · Cited by 22
Can Knowledge Graphs Reduce Hallucinations in LLMs? : A SurveyPreprint
Agrawal, G., Kumarage, T., Alghamdi, Z., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2311.07914 · Cited by 18 · see citing works
SmartPlay: A Benchmark for LLMs as Intelligent Agents
Wu, Y., Tang, X., Mitchell, T. M., & Li, Y. (2023) · arXiv · doi:10.48550/arxiv.2310.01557 · Cited by 6 · see citing works
Self-citationT-Eval: Evaluating the Tool Utilization Capability of Large Language Models Step by StepPreprint
Chen, Z., Du, W., Zhang, W., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2312.14033 · Cited by 3 · see citing works
Evaluating the Factuality of Large Language Models using Large-Scale Knowledge GraphsPreprint
Liu, X., Wu, F., Xu, T., et al. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2404.00942 · Cited by 1 · see citing works
OpenAI, Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.08774
Flagged first, then by citations
Research works printed but not linked: 14
- Jinheon Baek, Nirupama Chandrasekaran, Silviu Cucerzan, Allen Herring, and Sujay Kumar Jauhar. 2024. Knowledge-augmented large language models for personalized contextual query suggestion. In Pro- ceedings of the ACM on Web Conference 2024, WWW 2024, Singapore, May 13-17, 2024, pages 3355-3366. ACM. · DOI
- Prateek Chhikara, Jiarui Zhang, Filip Ilievski, Jonathan Francis, and Kaixin Ma. 2023. Knowledge-enhanced agents for interactive text games. In Proceedings of the 12th Knowledge Capture Conference 2023, K- CAP 2023, Pensacola, FL, USA, December 5-7, 2023, pages 157-165. ACM. · DOI · Paper page
- Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Samual Stevens, Boshi Wang, Huan Sun, and Yu Su. 2023. Mind2web: Towards a generalist agent for the web. In NeurIPS.
- Wenlong Huang, Pieter Abbeel, Deepak Pathak, and Igor Mordatch. 2022. Language models as zero-shot planners: Extracting actionable knowledge for em- bodied agents. In International Conference on Ma- chine Learning, ICML 2022, 17-23 July 2022, Balti- more, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pages 9118-9147.
- PMLR.
- Junling Liu, Peilin Zhou, Yining Hua, Dading Chong, Zhongyu Tian, Andrew Liu, Helin Wang, Chenyu You, Zhenhua Guo, Lei Zhu, and Michael Lingzhi Li. 2023a. Benchmarking large language models on cmexam -A comprehensive chinese medical exam dataset. In NeurIPS.
- OpenAI. 2023b. Gpt-4 technical report. arXiv preprint arXiv:2303.08774. · arXiv
- Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettle- moyer, Nicola Cancedda, and Thomas Scialom. 2023. Toolformer: Language models can teach themselves to use tools. In NeurIPS.
- Karthik Valmeekam, Matthew Marquez, Alberto Olmo Hernandez, Sarath Sreedharan, and Subbarao Kamb- hampati. 2023. Planbench: An extensible benchmark for evaluating large language models on planning and reasoning about change. In NeurIPS.
- Lei Wang, Chen Ma, Xueyang Feng, Zeyu Zhang, Hao Yang, Jingsen Zhang, Zhiyuan Chen, Jiakai Tang, Xu Chen, Yankai Lin, Wayne Xin Zhao, Zhewei Wei, and Jirong Wen. 2024. A survey on large language model based autonomous agents. Frontiers Comput. Sci., 18(6):186345. · DOI · Paper page
- Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. 2022. Chain-of-thought prompt- ing elicits reasoning in large language models. In NeurIPS.
- Songhua Yang, Hanjie Zhao, Senbin Zhu, Guangyu Zhou, Hongfei Xu, Yuxiang Jia, and Hongying Zan. 2024b. Zhongjing: Enhancing the chinese medi- cal capabilities of large language model through ex- pert feedback and real-world multi-turn dialogue. In AAAI, pages 19368-19376. AAAI Press. · DOI
- Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. 2023a. Tree of thoughts: Deliberate problem solving with large language models. In NeurIPS.
- Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R. Narasimhan, and Yuan Cao. 2023b. React: Synergizing reasoning and acting in language models. In ICLR. OpenReview.net.
Cited by
Sorted by citations
Evaluation and Benchmarking of LLM Agents: A Survey
Mohammadi, M., Li, Y., Lo, J., & Yip, W. (2025) · Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.2 · doi:10.1145/3711896.3736570 · Cited by 74 · see citing works
Self-citationKapoor, S., Stroebl, B., Siegel, Z. S., Nadgir, N., & Narayanan, A. (2024) · arXiv · doi:10.48550/arxiv.2407.01502 · Cited by 13 · see citing works
$τ^2$-Bench: Evaluating Conversational Agents in a Dual-Control Environment
Barres, V., Dong, H., Ray, S., Si, X., & Narasimhan, K. (2025) · arXiv · doi:10.48550/arxiv.2506.07982 · Cited by 1 · see citing works
“Existing benchmarks for conversational AI agents are designed to test their abilities to communicate effectively with a user and perform the right sequence of actions to solve tasks [26, 14, 23, 19].”
- Authors
- Xiao, R., Ma, W., Wang, K., Wu, Y., Zhao, J., Wang, H., Huang, F., & Li, Y.
- Affiliations
- Zhejiang University (CN)
- Alibaba Group (China) (CN)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2024
- Pages
- 10883-10900
- arXiv
- 2406.14884
- OpenAlex
- W4404792846
- Cited by
- 15
- References
- 21
- Citations by year
- 2026: 10 · 2025: 5
- FWCI
- 4.33
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- Fundamental Research Funds for the Central Universities
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › AI-based Problem Solving and Planning
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- LLM agents, benchmarking, agent planning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
