CoChat
Sign Up
PreprintCited by 13AI-based Problem Solving and PlanningSoftware Testing and Debugging TechniquesReinforcement Learning in Robotics

AJ-Bench: Benchmarking Agent-as-a-Judge for Environment-Aware Evaluation

Open access
Shi, Wentao, Wang, Yu, Zhao, Yuyang, Chen, Yuxin, Feng, Fuli, Hao, Xueyuan, Su, Xi, Gu, Qi +3 more·2026·arXiv
DOI 10.48550/arxiv.2604.18240arXiv 2604.18240
Open this paper in CoChat — free account

Abstract

As reinforcement learning continues to scale the training of large language model-based agents, reliably verifying agent behaviors in complex environments has become increasingly challenging. Existing approaches rely on rule-based verifiers or LLM-as-a-Judge models, which struggle to generalize beyond narrow domains. Agent-as-a-Judge addresses this limitation by actively interacting with environments and tools to acquire verifiable evidence, yet its capabilities remain underexplored. We introduce a benchmark AJ-Bench to systematically evaluate Agent-as-a-Judge across three domains-search, data systems, and graphical user interfaces-comprising 155 tasks and 516 annotated trajectories. The benchmark comprehensively assesses judge agents' abilities in information acquisition, state verification, and process verification. Experiments demonstrate consistent performance gains over LLM-as-a-Judge baselines, while also revealing substantial open challenges in agent-based verification. Our data and code are available at https://aj-bench.github.io/.

AI snapshot

Written from the paper’s full text.

TL;DR

AJ-Bench is introduced as a benchmark for evaluating Agent-as-a-Judge systems that actively interact with environments and tools to verify agent trajectories across search, data systems, and GUI domains. Experiments show Agent-as-a-Judge consistently outperforms LLM-as-a-Judge, but absolute performance remains unsaturated.

Key findings

  • AJ-Bench contains 155 tasks, 516 annotated trajectories, and 60 tools across Search, DS, and GUI domains.
  • Agent-as-a-Judge consistently outperforms LLM-as-a-Judge when built on the same base model, with an average improvement of 0.13 in F1; gpt-5-mini-low improves from 59.00 to 72.41 overall (+13.41), and deepseek-v3.2 improves from 64.49 to 77.34 (+12.85).
  • Despite gains, Agent-as-a-Judge is not saturated, with an average F1 score of 0.72; ablations show higher reasoning effort does not necessarily help, increasing interaction turns consistently improves F1, and mixed multimodal inputs do not uniformly improve GUI performance.
  • Failure-mode analysis reports that misinterpretation of tool outputs and incorrect reasoning despite correct evidence are prominent error categories across domains.

Methods

  • Benchmark construction: tasks are curated from Mind2Web2 and WideSearch for Search, MCPMark Filesystem and Postgres for DS, and OSWorld PowerPoint/Word/Excel for GUI; trajectories come from existing benchmarks and LLM regeneration, followed by human verification.
  • Labels are binary (1/0), assigned at item level in Search and trajectory level in DS and GUI; evaluation uses F1 between judge predictions and ground-truth annotations.
  • Agent-as-a-Judge is implemented on MCPMark, with environments replayed to final states for interaction; DS tasks are replayed locally and GUI tasks on isolated AWS instances. Main experiments compare many LLM-as-a-Judge models against Agent-as-a-Judge using gpt-5-mini-low and deepseek-v3.2, averaged over three runs.

Limitations

  • The paper states that most AJ-Bench tasks are adapted from existing benchmarks rather than created entirely from scratch, and that future work should construct more tasks independently and scale the data generation pipeline.
  • The paper states that in the Search domain, Agent-as-a-Judge relies on external web environments, so network instability may affect evaluation reliability.
  • Evident from the text: Agent-as-a-Judge experiments are limited to two representative models due to budget constraints, and some subdomain results have small sample sizes and wider confidence intervals; the GUI domain also uses a custom adapter, making the system sensitive to tool-formatting instructions.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

DataCite · 10.48550/arxiv.2604.18240

Cites

50 of 50 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

6 of 17 cited sources checked · 1 supported · 1 flagged · accuracy not yet measured

  • Premisecitation
    § 1 Introduction · p. 1

    Recent studies primarily rely on rule-based verification, where agent trajectories are evaluated against predefined rules.

    “Recent studies primarily rely on rule-based verification (Shao et al., 2024; Mroueh, 2025), where agent trajectories are evaluated against predefined rules (Wei et al., 2025).”
    Quote in the paperSubstantial claim3 of 3 sources checked
    • cites Shao et al., 2024 source

      The Shao et al. citation attaches to the claim that recent studies primarily rely on rule-based verification. Section 5.2.3 instead says that, in RL, the reward function is usually a neural reward model. The separate claim about evaluating agent trajectories is cited to Wei et al., not Shao et al.; no page number is available in our copy. (LLM, conf=0.83)

      Checked by a reasoning model — may be wrong.

      Experimental — accuracy not yet measured
    • cites Mroueh, 2025 source

      The Introduction (p. 2) describes several rule-based checks used for verifiable rewards, including answer matching, code tests, and formatting checks. It also includes LLM-as-judge verification and does not establish that recent studies primarily rely on rule-based methods. The separate claim about evaluating agent trajectories is cited to Wei et al., not Mroueh. (LLM, conf=0.82)

      Experimental — accuracy not yet measured
    • cites Wei et al., 2025 source

      Section 4.2 (p. 4) refers to rule-based evaluation, and the Introduction (p. 2) says rewards in the prior RL setup are derived entirely from the final action. The copy does not establish that whole agent trajectories are evaluated against predefined rules; it describes outcome-based action rewards instead. (LLM, conf=0.84)

      Experimental — accuracy not yet measured
  • Premisecitation
    § 1 Introduction · p. 1

    Rule-based verification methods do not generalize to complex, realistic settings where handcrafted rules are insufficient.

    “While effective for narrowly scoped tasks, such methods do not generalize to complex, realistic settings (e.g., scientific hypothesis verification or essay-level fact checking), where handcrafted rules are insufficient (Huang et al., 2025).”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites Huang et al., 2025 source

      The Introduction says that real-world scientific hypotheses are often abstract statements that are difficult to evaluate directly and must be translated into measurable implications. That supports the example of scientific hypothesis verification as a complex task, but our copy does not establish that rule-based methods fail to generalize, that handcrafted rules are insufficient, or that the same applies to essay-level fact checking. (LLM, conf=0.76)

      Experimental — accuracy not yet measured
  • Findingown data
    § 1 Introduction · p. 2

    AJ-Bench consists of 155 tasks.

    “The benchmark covers three domains-search, data system (DS), and graphical user interface (GUI)-and consists of 155 tasks spanning a wide range of complex agent behaviors.”
    • Also stated in: § 3.4 Dataset Statistics · p. 6 · plain
      “Following the construction pipeline described in §3.1-§3.3, we obtain a total of 155 tasks, 516 trajectories, and 60 different tools.”
    Quote in the paperNot a substantial claim
  • Findingown data
    § 1 Introduction · p. 2

    AJ-Bench contains 516 trajectories annotated with binary (positive and negative) labels.

    “We further collect 516 trajectories annotated with binary (positive and negative) labels.”
    • Also stated in: § 3.4 Dataset Statistics · p. 6 · plain
      “Following the construction pipeline described in §3.1-§3.3, we obtain a total of 155 tasks, 516 trajectories, and 60 different tools.”
    Quote in the paperSubstantial claim
  • Findingown data
    § 1 Introduction · p. 2

    Agent-as-a-Judge's absolute performance remains not yet saturated, with an average F1 score of 0.72.

    “However, its absolute performance remains not yet saturated, with an average F1 score of 0.72, leaving substantial room for further improvement.”
    Quote in the paperNot a substantial claim
  • May combine two claims. 29
  • Classified as probably not a substantial claim. 40

  • May combine two claims.

    Premiseown data
    § 1 Introduction · p. 2

    Existing LLM-as-a-Judge benchmarks (RewardBench, RM-Bench, JudgeBench, AgentRewardBench) are multi-domain but not environment-aware and do not allow agentic interaction.

    “| RewardBench (Lambert et al., 2025) | LLM-as-a-Judge | ✓ | ✗ | ✗ |”
    • Also stated in: § 1 Introduction · p. 2 · plain
      “| AgentRewardBench (Men et al., 2025) | LLM-as-a-Judge | ✓ | ✗ | ✗ |”
    Quote in the paperSubstantial claim2 of 2 sources checked
    • cites Lambert et al., 2025 source
    • cites Men et al., 2025 source

      Section 3 describes Agent-RewardBench as evaluating collected responses across perception, planning, and safety, which supports the multi-domain characterization and suggests an offline evaluation rather than live agent interaction. But the Related Work section says it tests web grounding and embodied spatial perception, so the claim that Agent-RewardBench is not environment-aware is not supported. The supplied table row names RewardBench, not Agent-RewardBench; this paper does not establish the stated properties of every benchmark in the claim. (LLM, conf=0.85)

      Experimental — accuracy not yet measured

  • May combine two claims.

    Findingown data
    § 1 Introduction · p. 2

    AJ-Bench is multi-domain, environment-aware, and allows agentic interaction.

    “| AJ-Bench | Agent-as-a-Judge | ✓ | ✓ | ✓ |”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § 4.3.2 Interaction Turns Ablation · p. 7

    Word and PPT tasks benefit more from extended interactions, indicating a greater reliance on iterative information gathering.

    “Word and PPT tasks benefit more from extended interactions, indicating a greater reliance on iterative information gathering.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingnone given
    § 1 Introduction · p. 2

    AJ-Bench is the first comprehensive benchmark for evaluating Agent-as-a-Judge systems, enabling rich interaction with environments and systematic assessment of their judgement capabilities.

    “We introduce the first comprehensive benchmark AJ-Bench for evaluating Agent-as-a-Judge systems, enabling rich interaction with environments and systematic assessment of their judgement capabilities.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Findingnone given
    § 3 Benchmark Construction · p. 4

    The benchmark emphasizes three core verification dimensions: information acquisition, state verification, and process verification.

    “The benchmark emphasizes three core verification dimensions: information acquisition, state verification, and process verification.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § 4.3.1 Thinking Ablation · p. 7

    For gpt-5-mini, the medium reasoning effort setting generally outperforms the low setting, while the high setting does not consistently outperform medium.

    “As shown in Table 4, across all three domains, we observe that for gpt-5-mini , the medium setting generally outperforms the low setting, while the high setting does not consistently outperform medium .”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § 1 Introduction · p. 2

    Agent-as-a-Judge consistently outperforms LLM-as-a-Judge, achieving an average improvement of 0.13 in F1.

    “Agent-asa-Judge consistently outperforms LLM-as-a-Judge, achieving an average improvement of 0.13 in F1.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Findingnone given
    § 3 Benchmark Construction · p. 4

    AJ-Bench is designed to evaluate a model's ability to leverage external tools when verifying agent trajectories.

    “AJ-Bench is designed to evaluate a model's ability to leverage external tools when verifying agent trajectories.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingreasoning
    § 1 Introduction · p. 2

    The collected tasks and trajectories jointly assess key judging capabilities: information acquisition via external search, state verification through tool-assisted interaction with environments, and process verification by inspecting critical actions and execution steps.

    “The collected tasks and trajectories jointly assess key judging capabilities, including (i) information acquisition via external search, (ii) state verification through tool-assisted interaction with environments, and (iii) process verification by inspecting critical actions and execution steps.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § 1 Introduction · p. 2

    Judge agents still show imperfect performance in evaluating complex, multi-step behaviors, suggesting substantial headroom for future improvement.

    “Experiments across three representative domains indicate that, despite their advantages, judge agents still show imperfect performance in evaluating complex, multi-step behaviors, suggesting substantial headroom for future improvement.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § 4.3.2 Interaction Turns Ablation · p. 7

    Increasing the interaction budget consistently improves F1 scores across all tasks, with the most pronounced gains observed for smaller budgets.

    “As shown in Figure 4, increasing the interaction budget consistently improves F1 scores across all tasks, with the most pronounced gains observed for smaller budgets.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data
    § 1 Introduction · p. 2

    AJ-Bench covers three domains: search, data system (DS), and graphical user interface (GUI).

    “The benchmark covers three domains-search, data system (DS), and graphical user interface (GUI)-and consists of 155 tasks spanning a wide range of complex agent behaviors.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § 5 Conclusions · p. 8

    The Agent-as-a-Judge baseline demonstrates strong performance and consistently outperforms LLM-as-a-Judge.

    “Our simple yet effective Agent-as-a-Judge baseline demonstrates strong performance and consistently outperforms LLM-as-a-Judge”
    • Also stated in: § A.4 Agent Framework Ablation · p. 12 · boosted
      “Currently, our Agent-as-a-Judge is built on MCPMark and demonstrates strong performance compared to LLM-as-a-Judge.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data
    § Correct evidence, wrong reasoning · p. 16

    The judge explicitly reasoned through both interpretations, even noting that 'if a file has duplicate content, it's not unique', which is the verify script's logic.

    “The judge explicitly reasoned through both interpretations, even noting *“if a file has duplicate content, it's not unique”* - which is the verify script's logic.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data
    § Correct evidence, wrong reasoning · p. 16

    The judge discussed both possible interpretations: the strict interpretation (all files with duplicate content go to duplicates/) and the lenient interpretation (keep one copy in root, move extras to duplicates/).

    “Discussed both possible interpretations: - Strict: ALL files with duplicate content go to ` duplicates/ ` (correct per verify script) - Lenient: keep one copy in root, move extras to ` duplicates/ `”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data
    § Failed to call tools · p. 15

    The judge used only 4 turns and 608 output tokens before the execution ended.

    “Only used 4 turns and 608 output tokens before the execution ended”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data
    § Misinterpreted tool output · p. 15

    The judge never examined whether the fp2 layer dimensions were correct.

    “Never examined whether the ` fp2 ` layer dimensions were correct”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data
    § Failed to call tools · p. 15

    The judge explicitly stated that the previous agent did not create the required file.

    “The judge explicitly stated: *“The previous agent did not create the required file.”*”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data
    § Failed to call tools · p. 15

    The verdict recorded for this case was PASS.

    “**Verdict recorded**: PASS”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data
    § Correct evidence, wrong reasoning · p. 16

    The judge verified 7 duplicate pairs by reading and hashing all files.

    “Verified 7 duplicate pairs by reading and hashing all files”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Findingown data
    § Failed to call tools · p. 15

    In the task to find a specific math paper and rename it to answer.html, the agent read 51 paper files but never renamed any file, and no answer.html was created.

    “**What the agent did**: Read 51 paper files but **never renamed any file** - no ` answer.html ` was created.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data
    § Correct evidence, wrong reasoning · p. 16

    The strict interpretation (all files with duplicate content go to duplicates/) is correct according to the verify script.

    “Strict: ALL files with duplicate content go to ` duplicates/ ` (correct per verify script)”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data
    § Correct evidence, wrong reasoning · p. 15

    The task was to find all files with duplicate content and move them to a duplicates/ directory.

    “**Task**: Find all files with duplicate content and move them to a ` duplicates/ ` directory.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Findingown data
    § Failed to call tools · p. 15

    In this case, the judge's own reasoning correctly concluded FAIL, but the recorded verdict was PASS.

    “The judge's own reasoning correctly concluded FAIL, but the recorded verdict was PASS.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Findingown data
    § Failed to call tools · p. 15

    The judge searched for answer.html and confirmed it does not exist.

    “Searched for ` answer.html ` -confirmed it does **not exist**”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingnone given
    § Correct evidence, wrong reasoning · p. 16

    The judge concluded that 'The agent's interpretation is defensible and consistent.'

    “But then concluded: > *“The agent's interpretation is defensible and consistent.”*”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data
    § Misinterpreted tool output · p. 15

    The judge traced the flow of set abstraction layers (sa1 → sa2 → sa3).

    “Traced the flow of set abstraction layers (sa1 → sa2 → sa3)”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data
    § Correct evidence, wrong reasoning · p. 16

    The judge confirmed one copy of each duplicate pair remained in the root directory.

    “Confirmed one copy of each duplicate pair remained in root directory”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data
    § Correct evidence, wrong reasoning · p. 16

    The verify script (lines 78-88) explicitly checks that zero files from any duplicate group remain in the original directory.

    “The verify script (lines 78-88) explicitly checks that **zero** files from any duplicate group remain in the original directory.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingnone given
    § 5 Conclusions · p. 8

    AJ-Bench covers three domains, Search, DS, and GUI, comprising 155 tasks and 516 trajectories.

    “AJ-Bench covers three domains, Search, DS, and GUI, comprising 155 tasks and 516 trajectories for comprehensive Agent-as-a-Judge evaluation.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data
    § Failed to call tools · p. 15

    The task was to find a specific math paper and rename it to answer.html.

    “**Task**: Find a specific math paper and rename it to ` answer.html ` .”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given
    § 4.1 Setup · p. 6

    Results reported in Table 3 are averaged over three runs.

    “Results reported in Table 3 are averaged over three runs.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductreasoning
    § 3.2.2 DS Domain · p. 5

    Trajectories associated with the same task are sourced from diverse model architectures and normalized into a consistent template format to mitigate potential bias from model-specific output styles.

    “To mitigate the potential bias introduced by model-specific output styles, we ensure that trajectories associated with the same task are sourced from diverse model architectures and subsequently normalized into a consistent template format.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Study design and conductnone given
    § 3.2 Trajectory Collection · p. 4

    All constructed trajectories are subjected to thorough human verification to ensure correctness and quality.

    “All constructed trajectories are subsequently subjected to thorough human verification to ensure correctness and quality.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Premisecitation
    § 3.1.2 DS Domain · p. 4

    The DS domain is constructed from two representative MCPMark subcategories, Filesystem and Postgres, which involve manipulating file structures and database records.

    “We construct the DS domain from two representative MCPMark (Wu et al., 2025) subcategories, Filesystem and Postgres, which involve manipulating file structures and database records.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Wu et al., 2025

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductown data
    § 3.3.1 Search Domain · p. 5

    For the WideSearch portion, labels are derived via majority voting across six models.

    “For the WideSearch portion, labels are obtained using the official WideSearch codebase. Specifically, labels are derived via majority voting across six models: gpt-4.1-2025-04-14 , gpt5-2025-08-07 , o4-mini-2025-04-16 , claude-sonnet4-20250514 , gemini-2.5-pro-preview-06-05 , and grok-3 .”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: Majority voting across six models produces reliable ground-truth labels for the WideSearch portion.

  • Probably not a substantial claim: may not pass S4 (based). May combine two claims.

    Premisenone given
    § 3.3.3 GUI Domain · p. 6

    These scripts are inherently limited in their ability to capture all execution details and edge cases, which may lead to mislabeling.

    “However, these scripts are inherently limited in their ability to capture all execution details and edge cases, which may lead to mislabeling.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Study design and conductreasoning
    § 3.2.3 GUI Domain · p. 5

    Trajectories are deliberately selected so that successful executions may involve many steps while failures terminate after relatively few steps, to decouple task success from trajectory length.

    “To mitigate potential bias arising from differences in trajectory length, where successful trajectories typically contain fewer steps than failed ones, we deliberately select trajectories in which successful executions may involve many steps while failures terminate after relatively few steps. This strategy helps decouple task success from trajectory length.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Study design and conductnone given
    § 3.2.2 DS Domain · p. 5

    A comprehensive manual quality check is performed to discard incomplete or noisy samples.

    “We further perform a comprehensive manual quality check to discard incomplete or noisy samples.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductreasoning
    § 3.3.2 DS domain · p. 5

    A manual validation pass is performed to correct potential misjudgements and maintain consistent annotation quality.

    “To further guarantee label correctness, we additionally perform a manual validation pass to correct potential misjudgements and maintain consistent annotation quality.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Study design and conductreasoning
    § 3.3.3 GUI Domain · p. 6

    To ensure label reliability, each GUI trajectory is manually inspected to verify its correctness.

    “To ensure label reliability, we therefore manually inspect each trajectory to verify its correctness.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationown data
    § 4.3.2 Interaction Turns Ablation · p. 7

    More information is retrieved through interaction with the environment, which notably aids evaluation.

    “This suggests that more information is retrieved through interaction with the environment, which notably aids evaluation.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Premisereasoning
    § 1 Introduction · p. 2

    Existing benchmarks fail to capture fundamental challenges faced by judge agents, including deciding when interaction is necessary, how to leverage tools effectively, and what constitutes sufficient and verifiable evidence for reliable judgement in open-ended environments.

    “Meanwhile, these benchmarks fail to capture the more fundamental challenges faced by judge agents, including deciding when interaction is necessary, how to leverage tools effectively, and what constitutes sufficient and verifiable evidence for reliable judgement in open-ended environments.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Study design and conductnone given
    § 3.2.3 GUI Domain · p. 5

    Trajectories are sampled from a diverse set of models to account for model heterogeneity.

    “Furthermore, we account for model heterogeneity by sampling trajectories from a diverse set of models.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § 3.3.3 GUI Domain · p. 6

    In the GUI domain, labels in OSWorld are initially assigned using rule-based scripts that compare execution trajectory outputs against golden files for office tasks.

    “In the GUI domain, labels in OSWorld are initially assigned using rule-based scripts that compare execution trajectory outputs against golden files for office tasks.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites OSWorld

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Premisecitation
    § 1 Introduction · p. 1

    LLM-as-a-Judge approaches have been explored, but their judgements are ultimately grounded in surface-level textual signals.

    “In parallel, LLM-as-a-Judge approaches have been explored, but their judgements are ultimately grounded in surface-level textual signals (Li et al., 2025; Gu et al., 2025).”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites Li et al., 2025

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Gu et al., 2025

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductown data
    § 3.2.2 DS Domain · p. 5

    For each task, up to three successful trajectories and three failed ones are retained, resulting in a balanced dataset.

    “For each task, we retain up to three successful trajectories and three failed ones, resulting in a balanced and high-quality dataset tailored for evaluating judge agents.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisereasoning
    § 3.1.3 GUI Domain · p. 4

    The GUI domain tasks are constructed from three office categories, PowerPoint, Word, and Excel, which require precise execution positions and carefully planned action sequences and thus remain challenging for current agents.

    “Specifically, we construct tasks from three office categories, PowerPoint, Word, and Excel, which require precise execution positions and carefully planned action sequences and thus remain challenging for current agents.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation
    § 1 Introduction · p. 2

    DevAI is an Agent-as-a-Judge benchmark that is environment-aware and allows agentic interaction but is not multi-domain.

    “| DevAI (Zhuge et al., 2025) | Agent-as-a-Judge | ✗ | ✓ | ✓ |”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Zhuge et al., 2025

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Classified as probably not a substantial claim.

    Study design and conductnone given
    § 3.3.1 Search Domain · p. 5

    Labels for the Mind2Web2 portion are obtained through manual annotation, where annotators assign a scoring rubric and corresponding rubric-level labels to each response.

    “Labels for the Mind2Web2 portion are obtained through manual annotation, where annotators assign a scoring rubric and corresponding rubric-level labels to each response.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Study design and conductnone given
    § 3.5 Environment Construction · p. 6

    To support interactive evaluation, the final environment state of each task is replayed, enabling the agent to interact with a live environment and actively acquire additional contextual information beyond static trajectories.

    “To support interactive evaluation, we replay the final environment state of each task, enabling the agent to interact with a live environment and actively acquire additional contextual information beyond static trajectories.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Replaying the final environment state of each task faithfully reconstructs the original environment for evaluation.

  • May combine two claims.

    Interpretationown data
    § Misinterpreted tool output · p. 15

    The judge was distracted by the agent's red-herring edit and could not independently determine what was wrong.

    “It was distracted by the agent's red-herring edit and could not independently determine what was wrong.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The ground truth bug in VoteNet's backbone_module.py is that self.fp2 = PointnetFPModule(mlp=[256,256,256]) should be mlp=[512,256,256] (input channels should be 256+256=512).

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductreasoning
    § Limitations · p. 9

    Instability in network connectivity may affect evaluation reliability in the search domain.

    “In the search domain, Agent-as-a-Judge relies on interactions with external web environments. As a result, instability in network connectivity may affect evaluation reliability.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Interpretationreasoning
    § Correct evidence, wrong reasoning · p. 16

    The judge identified the correct rule but chose not to apply it.

    “The judge identified the correct rule but chose not to apply it.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Interpretationnone given
    § 5 Conclusions · p. 8

    AJ-Bench is the first benchmark specifically designed for Agent-as-a-Judge evaluation.

    “introduce AJ-Bench as the first benchmark specifically designed for this purpose.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: Existing benchmarks lack systematic evaluation for Agent-as-a-Judge.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given
    § Limitations · p. 9

    Most tasks in AJ-Bench are adapted from existing benchmarks through modification rather than being created entirely from scratch.

    “Most tasks in AJ-Bench are adapted from existing benchmarks through modification rather than being created entirely from scratch.”
    Quote in the paperNot a substantial claim
56 of 56

  • The paper relies on this without support.

    Replaying the final environment state of each task faithfully reconstructs the original environment for evaluation.

    “To support interactive evaluation, we replay the final environment state of each task, enabling the agent to interact with a live environment and actively acquire additional contextual information beyond static trajectories.”

  • The paper relies on this without support.

    Majority voting across six models produces reliable ground-truth labels for the WideSearch portion.

    “Specifically, labels are derived via majority voting across six models: gpt-4.1-2025-04-14 , gpt5-2025-08-07 , o4-mini-2025-04-16 , claude-sonnet4-20250514 , gemini-2.5-pro-preview-06-05 , and grok-3 .”

  • The paper relies on this without support.

    The selected tasks and trajectories are representative of complex agent behaviors in open-ended environments.

    “The benchmark covers three domains-search, data system (DS), and graphical user interface (GUI)-and consists of 155 tasks spanning a wide range of complex agent behaviors.”

  • The paper relies on this without support.

    The F1 score between judge predictions and ground-truth annotations is an appropriate measure of judge capability.

    “Judge agents are evaluated using the F1 score between their predictions and the ground-truth annotations.”

  • The paper relies on this without support.

    The default configurations of models are fair for comparison.

    “Unless explicitly stated otherwise, all models are evaluated with their default configurations (e.g., temperature and reasoning effort).”

  • The paper relies on this without support.

    The environment replay supports concurrent evaluation without interference.

    “In both the DS and GUI domains, evaluation trajectories are replayed by sequentially executing the extracted action sequences in trajectories to reconstruct independent environments that support concurrent evaluation.”

  • The paper relies on this without support.

    Existing benchmarks lack systematic evaluation for Agent-as-a-Judge.

    “we identify a critical gap in existing benchmarks, namely the lack of systematic evaluation for Agent-as-a-Judge”

  • The paper relies on this without support.

    The selected subset of tasks (Wide, FileSystem, PPT) is representative of the three domains.

    “we conduct this analysis on a representative subset of tasks, selecting one subdomain from each of the three domains: Wide (Search), FileSystem (GUI), and PPT (DS).”

  • The paper relies on this without support.

    Agentic judging incurs substantially higher evaluation cost.

    “Because agentic judging incurs substantially higher evaluation cost”

  • The paper relies on this without support.

    The ground truth bug in VoteNet's backbone_module.py is that self.fp2 = PointnetFPModule(mlp=[256,256,256]) should be mlp=[512,256,256] (input channels should be 256+256=512).

    “**Ground truth bug**: ` self.fp2 = PointnetFPModule(mlp=[256,256,256]) ` should be ` mlp=[512,256,256] ` (input channels should be 256+256=512).”

  • The paper relies on this without support.

    OSWorld does not provide an official MCP server.

    “because OSWorld does not provide an official MCP 4”
11 of 11

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • Deepseek Postgres agentic recall conflicts with its false-negative rate

    Major

    The reported FNR of 6.38 implies recall of 93.62, not 72.60. The implied recall is also consistent with the agentic Postgres F1 score in Table 3.

    Reported 72.6 · recomputed 93.62

  • Deepseek Postgres baseline recall conflicts with its false-negative rate

    Major

    Recall must equal 100 minus the false-negative rate. The reported FNR of 11.35 implies recall of 88.65, not 60.32; the implied recall is also consistent with the Postgres F1 score in Table 3.

    Reported 60.32 · recomputed 88.65

  • FileSystem and PPT domains are reversed in the additional-model ablation

    Minor

    The appendix assigns FileSystem to GUI and PPT to DS, whereas the benchmark statistics place FileSystem under DS and PPT under GUI.

  • ReAct does not consistently outperform LLM-as-a-Judge

    Minor

    The appendix says both agentic frameworks consistently outperform LLM-as-a-Judge, but for gpt-5-mini-low on Wide, ReAct scores 51.13 versus the LLM-Judge's 60.84.

4 of 4

Cites

Flagged first, then by citations

Research works printed but not linked: 1

  1. Lei Li, Yekun Chai, Shuohuan Wang, Yu Sun, Hao Tian, Ningyu Zhang, and Hua Wu. 2024. Tool-augmented reward modeling. In ICLR . OpenReview.net.

Cited by

Sorted by citations

Authors
Shi, W., Wang, Y., Zhao, Y., Chen, Y., Feng, F., Hao, X., Su, X., Gu, Q., Su, H., Cai, X., & He, X.
Affiliations
University of Science and Technology of China (CN)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2026
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2604.18240
arXiv
2604.18240
OpenAlex
W7155091852
Cited by
0
References
35
Type
posted content
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › AI-based Problem Solving and Planning
  • Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
  • Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
Keywords
LLM agents, agent verification, benchmark, LLM-as-a-judge, information acquisition, state verification, process verification, search tasks, data systems, graphical user interface, large language models

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-16.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-16.

Cited sources are checked by gpt-6-sol.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review