The DOI matches its DataCite record; 3 of the 3 references it cites were found in a registry or an index.
WildClawBench: A Benchmark for Real-World, Long-Horizon Agent Evaluation
Abstract
Large language and vision-language models increasingly power agents that act on a user's behalf through command-line interface (CLI) harnesses. However, most agent benchmarks still rely on synthetic sandboxes, short-horizon tasks, mock-service APIs, and final-answer checks, leaving open whether agents can complete realistic long-horizon work in the runtimes where they are deployed. This work presents WildClawBench, a native-runtime benchmark of 60 human-authored, bilingual, multimodal tasks spanning six thematic categories. Each task averages roughly 8 minutes of wall-clock time and over 20 tool calls, and runs inside a reproducible Docker container hosting an actual CLI agent harness (OpenClaw, Claude Code, Codex, or Hermes Agent) with access to real tools rather than mock services. Grading is hybrid, combining deterministic rule-based checks, environment-state auditing of side effects, and an LLM/VLM judge for semantic verification. Across 19 frontier models, the best, Claude Opus 4.7, reaches only 62.2% overall under OpenClaw, while every other model stays below 60%, and switching harness alone shifts a single model by up to 18 points. These results show that long-horizon, native-runtime agent evaluation remains a far-from-resolved task for current frontier models. We release the tasks, code, and containerized tooling to support reproducible evaluation.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces
Merrill, M. A., Shaw, A. G., Carlini, N., Li, B., Raj, H., Bercovich, I., Shi, L., Shin, J. Y., Walshe, T., Buchanan, E. K., et al. (2026) · arXiv · doi:10.48550/arxiv.2601.11868 · Cited by 392 · cited 2×
“AgentBench [22] ⋄ ⋄ ⋄ Rule GAIA [25] ⋄ ⋄ Rule τ -bench [50] ⋄ State SWE-bench [17] ⋄ ⋄ Exec WebArena [59] ⋄ ⋄ State VisualWebArena [20] ⋄ State OSWorld [45] ⋄ Hybrid Terminal-Bench [24] ⋄ Exec PinchBench [18] ⋄ ⋄ Hybrid Claw-Eval [51] ⋄ Hybrid ClawBench [55] ⋄ Hybrid”
Self-citationAgentHarm: A Benchmark for Measuring Harmfulness of LLM Agents
Andriushchenko, M., Souly, A., Dziemian, M., Duenas, D., Lin, M., Wang, J., Hendrycks, D., Zou, A., Kolter, Z., Fredrikson, M., et al. (2024) · arXiv · doi:10.48550/arxiv.2410.09024 · Cited by 11
Self-citationAgent-SafetyBench: Evaluating the Safety of LLM Agents
Zhang, Z., Cui, S., Lu, Y., Zhou, J., Yang, J., Wang, H., & Huang, M. (2024) · arXiv · doi:10.48550/arxiv.2412.14470 · Cited by 6 · cited 1×
“ToolEmu [34] and Agent-SafetyBench [57] argue for trajectory-level reasoning, and Claw-Eval [51] demonstrates multi-channel evidence auditing with controlled error injection.”
Self-citation
Cited by
Sorted by citations
OSWorld 2.0: Benchmarking Computer Use Agents on Long-Horizon Real-World Tasks
Yuan, M., Zhou, Z., Xiong, X., Wu, W., Sun, J., Song, J., Cui, K., Wang, B., Wu, H., Li, Y., et al. (2026) · arXiv · doi:10.48550/arxiv.2606.29537 · Cited by 79
Claw-SWE-Bench: A Benchmark for Evaluating OpenClaw-style Agent Harnesses on Coding Tasks
Zheng, M., Han, K., Li, B., Xu, H., Tian, Y., He, W., Zhou, H., Guo, J., Hu, H., Ma, L., et al. (2026) · arXiv · doi:10.48550/arxiv.2606.12344
“These functions support models in tasks involving productivity tools, browser automation, computer use, and scientific assistance [4, 18].”
HarnessRisk: A Lifecycle-Oriented Benchmark for Agent Harness Safety
Bai, Y., Duan, J., Peng, J., Wu, X., Liu, S., Wang, S., & Chen, T. (2026) · arXiv · doi:10.48550/arxiv.2608.17597
Metadata
- Authors
- Ding, S., Dai, X., Xing, L., Ding, S., Liu, Z., JingYi, Y., Yang, P., Zhang, Z., Wei, X., Fang, X., Ma, Y., Duan, H., Shao, J., Wang, J., Lin, D., Chen, K., & Zang, Y.
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2026
- arXiv
- 2605.10912
- References
- 3
- Type
- Preprint
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
