AutomationBench
Open accessAbstract
Existing AI benchmarks for software automation rarely combine cross-application coordination, autonomous API discovery, and policy adherence. Real business workflows demand all three: a single task may span a CRM, inbox, calendar, and messaging platform - requiring the agent to find the right endpoints, follow a policy document, and write correct data to each system. To address this gap, we introduce AutomationBench, a benchmark for evaluating AI agents on cross-application workflow orchestration via REST APIs. Drawing on real workflow patterns from Zapier's platform, tasks span Sales, Marketing, Operations, Support, Finance, and HR domains. Agents must discover relevant endpoints themselves, follow layered business rules, and navigate environments with irrelevant and sometimes misleading records. Grading is programmatic and end-state only: whether the correct data ended up in the right systems. Even the best frontier models currently score below 10%. AutomationBench provides a challenging, realistic measure of where current models stand relative to the agentic capabilities businesses actually need.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Across AppWorld, τ2-bench, and AutomationBench, DAEDALUS improves mean success rates by up to 15.9 percentage points over a no-memory baseline.
“Across AppWorld, τ 2 -bench, and AutomationBench, DAEDALUS improves mean success rates by up to 15.9 percentage points and pass^5 by up to 2.2 × over a no-memory baseline, and is competitive with methods using training tasks, at a lower inference cost than most.”
From DAEDALUS: Bootstrapping Agent Memory from Self-Generated Tasks (2026) · cites this paper as Shepard & Salimans, 2026
The Results section reports AutomationBench scores for frontier models, not DAEDALUS or a no-memory comparison. Searches for DAEDALUS, 15.9 percentage points, and the claimed cross-benchmark result found no passage in our copy addressing that result. (LLM, conf=0.76)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
No references could be resolved yet.
Research works printed but not linked: 9
- S. Zhou, F. F. Xu, H. Zhu, X. Zhou, R. Lo, A. Sridhar, X. Cheng, T. Ou, Y. Bisk, D. Fried, U. Alon, G. Neubig. WebArena: A Realistic Web Environment for Building Autonomous Agents. ICLR, 2024.
- X. Deng, Y. Gu, B. Zheng, S. Chen, S. Stevens, B. Wang, H. Sun, Y. Su. Mind2Web: Towards a Generalist Agent for the Web. NeurIPS, 2023.
- T. Xie, D. Zhang, J. Chen, X. Li, S. Zhao, R. Cao, T. J. Hua, Z. Cheng, D. Shin, F. Lei, Y. Liu, Y. Xu, S. Zhou, S. Savarese, C. Xiong, V. Zhong, T. Yu. OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments. NeurIPS, 2024.
- Y. Qin, S. Liang, Y. Ye, K. Zhu, L. Yan, Y. Lu, Y. Lin, X. Cong, X. Tang, B. Qian, S. Zhao, L. Hong, R. Tian, R. Xie, J. Zhou, M. Gerstein, D. Li, Z. Liu, M. Sun. ToolLLM: Facilitating Large Language Models to Master 16000+ Real-World APIs. ICLR, 2024.
- M. Li, Y. Zhao, B. Yu, F. Song, H. Li, H. Yu, Z. Li, F. Huang, Y. Li. API- Bank: A Comprehensive Benchmark for Tool-Augmented LLMs. EMNLP, 2023.
- H. Trivedi, T. Khot, M. Hartmann, R. Manku, V. Dong, E. Li, S. Gupta, A. Sabharwal, N. Balasubramanian. AppWorld: A Controllable World of Apps and People for Benchmarking Interactive Coding Agents. ACL, 2024.
- S. Yao, N. Shinn, P. Razavi, K. Narasimhan. τ -bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains. arXiv preprint arXiv:2406.12045, 2024. · arXiv
- V. Barres, H. Dong, S. Ray, X. Si, K. Narasimhan. τ 2 -Bench: Evaluating Conversational Agents in a Dual-Control Environment. arXiv preprint arXiv:2506.07982, 2025. · arXiv
- Q. Shi, A. Zytek, P. Razavi, K. Narasimhan, V. Barres. τ -Knowledge: Evaluating Conversational Agents over Unstructured Knowledge. arXiv preprint arXiv:2603.04370, 2026. · arXiv
Cited by
Sorted by citations
Kimi K3: Open Frontier Intelligence
Kimi Team, Bai, T., Bai, Y., Bao, Y., C., M., Cai, J., Cai, X., Cao, P., Cao, Y., Chai, Z., et al. (2026) · arXiv · doi:10.48550/arxiv.2607.24653
“…: BrowseComp [133], DeepSearchQA [128], ResearchRubrics [107], Toolathlon-Verified [70], MCPMark-Verified [135], MCP-Atlas [11], AutomationBench [109], JobBench [71], GDPval-AA v2 [91], AA-Briefcase [8, 2], Agents’ Last Exam (ALE) [4, 117], APEX-Agents [129], OfficeQA Pro [88], SpreadsheetBench…”
DAEDALUS: Bootstrapping Agent Memory from Self-Generated Tasks
Edy, A., Conti, M., Xing, V., Allard, M.-A., Benhamdane, N., & Viaud, G. (2026) · arXiv · doi:10.48550/arxiv.2610.08048
“We run evaluations on three agentic benchmarks with different task types and environments: code-based app automation in AppWorld (Trivedi et al., 2024), conversational customer service in τ 2 -bench (Barres et al., 2026), and business workflows with SaaS applications in Au-tomationBench (Shepard & Salimans, 2026).”
- Authors
- Shepard, D., & Salimans, R.
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2026
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2604.18934
- OpenAlex
- W7155161300
- Cited by
- 0
- References
- 0
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Engineering › Industrial and Manufacturing Engineering › Robotic Process Automation Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Artificial Intelligence Applications
- Physical Sciences › Computer Science › Artificial Intelligence › AI-based Problem Solving and Planning
- Keywords
- AI agents, software automation, benchmark, business rules, agent evaluation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
