OSWorld 2.0: Benchmarking Computer Use Agents on Long-Horizon Real-World Tasks

It may state two separate claims as one.

GPT-5.5 is far more token-efficient than Claude Opus 4.8 yet plateaus near 13% binary completion.

“GPT-5.5 is far more token-efficient yet plateaus near 13%.”

• Also stated in: § 1 Introduction · plain

  “GPT-5.5, the most token-efficient agent, plateaus near 13% at under a fifth of Opus's token budget”

It may state two separate claims as one.

Leading agents average more than 300 steps per task on OSWORLD 2.0, against about 30 in OSWorld 1.0.

“leading agents average more than 300 steps per task, against about 30 in OSWorld 1.0.”

Probably not a substantial claim. The paper’s conclusions may not depend on it.

OSWORLD 2.0 scores the final state against fine-grained checkpoints averaging 27.25 per task.

“We score the final state against fine-grained checkpoints (averaging 27.25 per task)”

It may state two separate claims as one.

Under the primary binary-completion metric at 500 steps, Claude Opus 4.8 with maximum thinking and batched tool calls scores best but completes only 20.6% of tasks at a 54.8% partial score.

“Under our primary binary-completion metric at 500 steps, Claude Opus 4.8 with maximum thinking and batched tool calls scores best but still completes only 20.6% of tasks at a 54.8% partial score”

• Also stated in: § 1 Introduction · plain

  “Even the strongest configuration, Claude Opus 4.8 with maximum thinking and batched tool calls, completes only 20.6% of tasks under strict binary completion while reaching 54.8% partial score”

• Also stated in: § 3.2 Main Results · plain

  “The best 500-step configuration in Table 3, Claude Opus 4.8 with max thinking and the batched tool, reaches only 20.6% binary completion and 54.8% partial score, while GPT-5.5 reaches 13.0% and Claude Opus 4.7 reaches 18.2%.”

It may state two separate claims as one.

The median OSWORLD 2.0 task takes a skilled human about 1.6 hours of active operation, roughly 48× longer than OSWorld 1.0.

“The median task takes a skilled human about 1.6 hours of active operation, roughly 48× longer than OSWorld 1.0”

• Also stated in: § 2.2.1 Task Statistics · hedged

  “the median human operation time is approximately 1.6 hours, about 48× longer than the roughly two-minute median in OSWORLD 1.0”

• AssumptionRests on an assumption: The estimated human completion times recorded by the annotators accurately reflect the time a skilled human user would take to complete each task.

It may state two separate claims as one.

OSWORLD 2.0 tasks require an average of 318 tool calls with Claude Opus 4.7 using maximum thinking, compared with about 30 in OSWORLD 1.0.

“requires an average of 318 tool calls with Claude Opus 4.7 using maximum thinking, compared with about 30 in OSWORLD 1.0.”

It may state two separate claims as one.

Empirical agent difficulty broadly tracks human-predicted difficulty, except on tasks humans find easy, where perceptual and interactive demands keep most workflows hard for agents.

“Empirical agent difficulty broadly tracks human-predicted difficulty, except on tasks humans find easy, where perceptual and interactive demands keep most workflows hard for agents.”

• AssumptionRests on an assumption: The estimated human completion times recorded by the annotators accurately reflect the time a skilled human user would take to complete each task.

It may state two separate claims as one.

GPT-5.5 is the most token-efficient agent by a wide margin, reaching ∼14% binary reward at only ∼37K output tokens per task.

“GPT-5.5 is the most token-efficient agent by a wide margin, reaching ∼14% binary reward at only ∼37K output tokens per task”

It may state two separate claims as one.

The final model looks reasonable but fails on exact geometry.

“The final model looks reasonable but fails on exact geometry.”

It may state two separate claims as one.

Task 100 succeeds under either GUI operation or direct state editing because the underlying state is recoverable, while tasks 003 and 010 require exact final state or format and penalize styles that omit verification.

“task 100 succeeds under either GUI operation or direct state editing because the underlying state is recoverable, while tasks 003 and 010 require exact final state or format and penalize styles that omit verification”

It may state two separate claims as one.

Tasks humans find easy often remain hard for agents, so progress transfers only partially from horizon-based scaling laws.

“tasks humans find easy often remain hard for agents, so progress transfers only partially from horizon-based scaling laws.”

It may state two separate claims as one.

Completion rates range from 4.6% to 14.0% and partial-only rates from 50.0% to 67.6%, with a median score of 0.44 on non-zero runs.

“Completion rates range from 4.6% to 14.0% and partial-only rates from 50.0% to 67.6%, with a median score of 0.44 on non-zero runs.”

It may state two separate claims as one.

Claude Opus 4.7 needs ∼190 turns for 18.2%, while the single-action Sonnet 4.6 and MiniMax M3 sit at 250 to 325 turns for under 10%.

“Claude Opus 4.7 needs ∼190 turns for 18.2%, while the single-action Sonnet 4.6 and MiniMax M3 sit far to the right at 250 to 325 turns for under 10%.”

It may state two separate claims as one.

Only 11.1% of human-easy tasks are also easy for agents, while 44.4% remain hard and another 44.4% are medium.

“only 11.1% of tasks are also easy for agents while 44.4% remain hard and another 44.4% are medium.”

• AssumptionRests on an assumption: The estimated human completion times recorded by the annotators accurately reflect the time a skilled human user would take to complete each task.

It may state two separate claims as one.

The same six-fold increase in tokens adds only ∼5 partial points, yet still separates 13% from 20% in the binary view.

“The same six-fold increase in tokens adds only ∼5 partial points here, yet still separates 13% from 20% in the binary view of Figure 1.”

It may state two separate claims as one.

Agents mainly fail in long-horizon tasks along five recurring dimensions: information grounding and tracking, perception-action timing, domain knowledge and workflow learning, verification and reflection, and long-horizon state drift, rather than lacking the basic ability to perform GUI actions or write code.

“Rather than lacking the basic ability to perform GUI actions or write code, agents mainly fail in long-horizon tasks along five recurring dimensions: information grounding and tracking, perception-action timing, domain knowledge and workflow learning, verification and reflection, and long-horizon state drift.”

It may state two separate claims as one.

By instruction and setup alone, 64.8% of OSWORLD 2.0 tasks require two or more apps or services; with rollout usage included, the share rises to 75.9%.

“By instruction and setup alone, 64.8% of tasks require two or more apps or services; with rollout usage included, the share rises to 75.9%.”

It may state two separate claims as one.

GPT-5.5 is stronger than Opus 4.7 on Visual-spatial Precision (51.2 vs. 43.9) and Multimodal Editing (47.0 vs. 44.0).

“GPT-5.5 is stronger on the visual and media phenomena, Visual-spatial Precision (51.2 vs. 43.9)and Multimodal Editing (47.0 vs. 44.0)”

It may state two separate claims as one.

In the FreeCAD reconstruction task (Task 103), agents read dimensions from an engineering drawing and generate a plausible support bracket, but do not keep a stable mapping from features to dimensions, primitives, and geometric checks.

“In the FreeCAD reconstruction task (Task 103; Figure 12, bottom row; Appendix H.1.2), agents read dimensions from an engineering drawing and generate a plausible support bracket, but do not keep a stable mapping from features to dimensions, primitives, and geometric checks.”

It may state two separate claims as one.

In the purchase-order task (Task 035), a new TeamChat message overrides an earlier rule while the agent is reading another request, and a later correction changes the budget and vendor constraints again.

“In the purchase-order task (Task 035; Figure 12, top row; Appendix H.2.2), a new TeamChat message overrides an earlier rule while the agent is reading another request, and a later correction changes the budget and vendor constraints again.”

It may state two separate claims as one.

MiniMax M3 mixes the two modes but has the highest churn rate among the four at 24%, leaving 45% of its runs at zero score.

“MiniMax M3 also mixes the two modes but has the highest churn rate among the four at 24%, leaving 45% of its runs at zero score.”

It may state two separate claims as one.

Agents often fail to verify task-critical properties and correct errors they have already noticed before submission.

“Agents often fail to verify task-critical properties and correct errors they have already noticed before submission.”

Probably not a substantial claim. The paper’s conclusions may not depend on it.

Research & Education and Creative Production together account for over 40% of the OSWORLD 2.0 benchmark.

“Research & Education and Creative Production together account for over 40% of the benchmark.”

It may state two separate claims as one.

The close button is visible in the screenshot, but by the time the click is issued the pop-up has shifted to a new location.

“The close button is visible in the screenshot, but by the time the click is issued the pop-up has shifted to a new location.”

It may state two separate claims as one.

GPT-5.5 and Claude Opus 4.7 achieve 20-24% binary completion accuracy on shorter tasks (under 45 minutes), but these success rates decline rapidly as workflows extend, and by the 137-163 minute bin no model exceeds 10% accuracy.

“GPT-5.5 and Claude Opus 4.7 achieve 20-24% binary completion accuracy on shorter tasks (under 45 minutes), but these success rates decline rapidly as workflows extend, and by the 137-163 minute bin no model exceeds 10% accuracy.”

It may state two separate claims as one.

76.3% of human-hard tasks are also hard for agents, and the share of agent-hard outcomes decreases monotonically as human difficulty drops, from 76.3% (Hard) to 57.4% (Medium) and 44.4% (Easy).

“76.3% of human-hard tasks are also hard for agents, and the share of agent-hard outcomes decreases monotonically as human difficulty drops, from 76.3% (Hard) to 57.4% (Medium) and 44.4% (Easy).”

• AssumptionRests on an assumption: The estimated human completion times recorded by the annotators accurately reflect the time a skilled human user would take to complete each task.

It may state two separate claims as one.

In streaming interfaces, the agent can correctly reason about the observed screenshot, but the screen changes before the action is executed.

“In streaming interfaces, the agent can correctly reason about the observed screenshot, but the screen changes before the action is executed.”

It may state two separate claims as one.

Reaching the first ∼14% costs only ∼37K tokens with GPT-5.5, while pushing to 18.2% takes ∼150K tokens with Claude Opus 4.7 and reaching 20.5% takes ∼225K with Claude Opus 4.8, or roughly 25 to 30K extra tokens per additional point.

“Reaching the first ∼14% costs only ∼37K tokens with GPT-5.5, which is a few thousand tokens per point, while pushing to 18.2% then takes ∼150K tokens with Claude Opus 4.7 and reaching 20.5% takes ∼225K with Claude Opus 4.8, or roughly 25 to 30K extra tokens for each additional point.”

Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.

A task may carry multiple challenge-phenomenon tags, so percentages in Table 2 sum to more than 100%.

“a task may carry multiple tags, so percentages in Table 2 sum to more than 100%.”

It may state two separate claims as one.

On OSWorld 2.0, the mean step budget consumed by agents scales with human-estimated duration: the 0-150 step bin shrinks from 20.8% on human-easy tasks to 0% on human-hard tasks, while the 301-450 step bin grows from 18.9% to 42.9%.

“On OSWORLD 2.0, the mean step budget consumed by agents does scale with human-estimated duration: the 0-150 step bin shrinks from 20.8% on human-easy tasks to 0% on human-hard tasks, while the 301-450 step bin grows from 18.9% to 42.9%.”

It may state two separate claims as one.

Every strong agent jumps into a tight band between 41% and 54% partial score, far above the 8 to 20% spread in binary completion.

“Every strong agent jumps into a tight band between 41% and 54%, far above the 8 to 20% spread in binary completion.”

It may state two separate claims as one.

Model-based evaluation contributes 11.53% of the total OSWORLD 2.0 score, and no task relies on it for more than 50%.

“Overall, model-based evaluation contributes 11.53% of the total score, and no task relies on it for more than 50%.”

It may state two separate claims as one.

Claude Opus 4.7 reaches 18.2% at around 150K tokens, while Claude Opus 4.8 reaches the best result on the benchmark, 20.5%, at around 225K tokens.

“Claude Opus 4.7 reaches 18.2% at around 150K tokens, while Claude Opus 4.8 reaches the best result on the benchmark, 20.5%, at around 225K.”

Probably not a substantial claim.

OSWORLD 2.0 tasks are designed to measure complete computer-use workflows rather than isolated interface operations.

“These tasks are designed to measure complete computer-use workflows rather than isolated interface operations.”

It may state two separate claims as one.

The largest mapped economic-value shares of the OSWORLD 2.0 task set are document preparation (23.8%), software and databases (17.8%), and finance and operations analysis (16.3%).

“The largest mapped shares are document preparation (23.8%), software and databases (17.8%), and finance and operations analysis (16.3%).”

• AssumptionRests on an assumption: The wage-bill-based GDP proxy validly estimates the economic coverage of the OSWORLD 2.0 task set.

Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.

Sonnet 4.6 is a stronger hybrid solver whose failures are typically exactness errors rather than complete breakdowns.

“Sonnet 4.6 is a stronger hybrid solver whose failures are typically exactness errors rather than complete breakdowns.”

It may state two separate claims as one.

OSWORLD 1.0 averages roughly 30 steps, while OSWORLD 2.0 requires more than 250 steps per task under the strongest evaluation setting.

“OSWORLD 1.0 averages roughly 30 steps, while OSWORLD 2.0 requires more than 250 steps per task under our strongest evaluation setting.”

Probably not a substantial claim. It may state two separate claims as one.

OSWORLD 2.0 tasks require agents to work over long horizons, move across applications and services, recover task-relevant state from realistic artifacts, and keep that state consistent until the final submission or deliverable.

“They require agents to work over long horizons, move across applications and services, recover task-relevant state from realistic artifacts, and keep that state consistent until the final submission or deliverable.”

It may state two separate claims as one.

Opus 4.7 spends under 2% on recovery and roughly 3% on repair, GPT-5.5 and Sonnet 4.6 allocate close to 3% to each, and MiniMax M3 keeps both below 5%.

“Opus 4.7 spends under 2% on recovery and roughly 3% on repair, GPT-5.5 and Sonnet 4.6 allocate close to 3% to each, and even MiniMax M3, the highest on this dimension, keeps both below 5%.”

It may state two separate claims as one.

Across the 216 evaluated task trajectories (108 tasks per model), agents extracted hidden application states in about 14% of tasks and bypassed user-visible interfaces in about 33% of tasks.

“across the 216 evaluated task trajectories (108 tasks per model), this primarily manifests as extracting hidden application states (about 14% of tasks) or bypassing user-visible interfaces (about 33% of tasks).”

It may state two separate claims as one.

Models with a committed style, programmatic in GPT-5.5 or balanced in Opus 4.7, achieve the lowest zero-score rates of 19% and 31%, whereas the highest-churn system also has the highest zero-score rate.

“Models with a committed style, programmatic in GPT-5.5 or balanced in Opus 4.7, achieve the lowest zero-score rates of 19% and 31%, whereas the highest-churn system also has the highest zero-score rate”

It may state two separate claims as one.

In MyPCBench, Claude Sonnet 4.6 averages 45.8 steps per task, while in OSWORLD 2.0 the stronger Claude Opus 4.7 still averages 318.4 steps per task.

“In MyPCBench, Claude Sonnet 4.6 averages 45.8 steps per task, while in OSWORLD 2.0 the stronger Claude Opus 4.7 still averages 318.4 steps per task.”

It may state two separate claims as one.

In the reimbursement task (Task 008), the agent reads the account-code rules, cross-checks bank charges, and submits the report.

“In the reimbursement task (Task 008; Figure 1; Appendix H.1.1), the agent reads the account-code rules, cross-checks bank charges, and submits the report.”

It may state two separate claims as one.

GPT-5.5 reaches ∼49.5% partial score at only ∼37K tokens, essentially tying Claude Opus 4.7, which needs ∼150K tokens for 48.9%, and coming within five points of Opus 4.8's best of 54.2% at ∼225K.

“GPT-5.5 reaches ∼49.5% at only ∼37K tokens, essentially tying Claude Opus 4.7, which needs ∼150K tokens for 48.9%, and coming within five points of Opus 4.8's best of 54.2% at ∼225K.”

It may state two separate claims as one.

Claude Opus 4.8 is the most token-expensive agent on the benchmark yet one of the cheapest in turns, reaching its benchmark-best 20.6% in only ∼103 turns because batched calls let it act several times per turn.

“Claude Opus 4.8 is the most token-expensive agent on the benchmark yet one of the cheapest in turns, reaching its benchmark-best 20.6% in only ∼103 turns because batched calls let it act several times per turn.”

It may state two separate claims as one.

Single-app OSWORLD 2.0 tasks require deep use of specialized creative, engineering, and scientific tools, or focused operation of a single web service.

“Singleapp tasks are still substantial: they require deep use of specialized creative, engineering, and scientific tools, or focused operation of a single web service.”

It may state two separate claims as one.

Agents may lose explicit instruction constraints, overlook information revealed during execution, or proceed under missing evidence when they should ask the user.

“Agents may lose explicit instruction constraints, such as required file formats or naming rules, overlook information revealed during execution, or proceed under missing evidence when they should ask the user.”

It may state two separate claims as one.

On the human-easy side, only 11.1% of tasks are also easy for agents, while 44.4% remain hard and another 44.4% are medium.

“on the human-easy side, where only 11.1% of tasks are also easy for agents while 44.4% remain hard and another 44.4% are medium.”

Probably not a substantial claim. The paper’s conclusions may not depend on it.

The instruction for Task 103 names no FreeCAD operations.

“The instruction names no FreeCAD operations.”

Probably not a substantial claim.

After finding the checkout logic, the agent types another javascript: URL that directly calls /api/task052/open-checkout with method: 'POST', credentials: 'include', and a JSON body containing hotel-paris-1, Le Meurice, and Deluxe Suite.

“Instead, after finding the checkout logic, the agent types another javascript: URL that directly calls /api/task052/open-checkout with method: 'POST', credentials: 'include', and a JSON body containing hotel-paris-1, Le Meurice, and Deluxe Suite.”

Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.

In the Claude Sonnet 4.6 medium trajectory, the agent completes the visible task with main score 1.0, but the safety check fails: the evaluator scans 11 repository files and reports credential_leak because wandb_api_key remains in the pushed repository.

“In the Claude Sonnet 4.6 medium trajectory, the agent completes the visible task with main score 1.0, but the safety check fails: the evaluator scans 11 repository files and reports credential_leak because wandb_api_key remains in the pushed repository.”

Probably not a substantial claim. It may state two separate claims as one.

After issuing the ASK_USER call (step 283), the agent presents a structured summary of completed steps, the $6,000 funding shortfall, and the portal's explicit rejection warning.

“After issuing the ASK_USER call (step 283), the agent presents a structured summary of completed steps, the $6,000 funding shortfall, and the portal's explicit rejection warning.”

Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.

The agent abandons Shotcut's GUI early in favour of a scriptable approach: it uses ffprobe to confirm the video metadata, extracts sampled keyframes at regular intervals to inspect spider positions visually, and writes a ffmpeg script that applies drawbox filters timed to cover the spider's location in each segment.

“It abandons Shotcut's GUI early in favour of a scriptable approach: it uses ffprobe to confirm the video metadata, extracts sampled keyframes at regular intervals to inspect spider positions visually, and writes a ffmpeg script that applies drawbox filters timed to cover the spider's location in each segment (Figure 45).”

Probably not a substantial claim. The paper’s conclusions may not depend on it.

The key dynamic of Task 035 is that the manager's requirements are not static: new messages arrive during the agent's execution that update constraints the agent has already internalized.

“The key dynamic is that the manager's requirements are not static: new messages arrive during the agent's execution that update constraints the agent has already internalized.”

It may state two separate claims as one.

In Task 008, Opus 4.7 carries much of the reimbursement evidence through a long GUI workflow, but exact per-diem and attachment details drift before submission.

“In Task 008 (Figure1; Appendix H.1.1), it carries much of the reimbursement evidence through a long GUI workflow, but exact per-diem and attachment details drift before submission.”

It may state two separate claims as one.

In fixed five-step windows, the most common activities are visual grounding (15.5%), tool-semantics reasoning (13.8%), and information extraction (12.8%), all outranking execution (10.1%) and verification (9.8%).

“In fixed five-step windows, the most common activities are visual grounding (15.5%), tool-semantics reasoning (13.8%), and information extraction (12.8%), all outranking execution (10.1%) and verification (9.8%).”

It may state two separate claims as one.

The 202 steps span qualitatively distinct reasoning phases: spatial interpretation of multi-view projections, API-level scripting decisions, iterative geometric refinement guided by visual comparison, and output verification.

“The 202 steps span qualitatively distinct reasoning phases: spatial interpretation of multi-view projections, API-level scripting decisions, iterative geometric refinement guided by visual comparison, and output verification.”

Probably not a substantial claim. The paper’s conclusions may not depend on it.

There is no written style guide for Task 004; the reference is the existing slides themselves.

“There is no written style guide; the reference is the existing slides themselves.”

It may state two separate claims as one.

MiniMax M3 and Qwen 3.7-Plus show the highest churn rates at 24% and 18%, which coincide with the highest zero-score rates.

“MiniMax M3 and Qwen 3.7-Plus show the highest churn rates at 24% and 18%, which coincide with the highest zero-score rates.”

Probably not a substantial claim.

A confirmation prompt then asks whether the user is sure they want to discard the LibreOffice document recovery data.

“A confirmation prompt then asks whether the user is sure they want to discard the LibreOffice document recovery data.”

It may state two separate claims as one.

All user-simulating models exceed 95% human-verified accuracy; Claude Sonnet 4.6 reaches 100%, supporting its selection as the user-simulating model.

“Table 12 shows that all models exceed 95% human-verified accuracy; Claude Sonnet 4.6 reaches 100%, supporting its selection as the user-simulating model.”

It may state two separate claims as one.

The agent sees both a credential file and a hard-coded W&B key, but its final checks focus on whether the project files, model outputs, slides, and GitLab repository are complete.

“The agent sees both a credential file and a hard-coded W&B key, but its final checks focus on whether the project files, model outputs, slides, and GitLab repository are complete.”

It may state two separate claims as one.

OSWORLD 2.0 contains 108 end-to-end tasks spanning everyday and professional work, and targets challenge phenomena that are common in real workflows, including dynamic environments, streaming interaction, proactive interaction, cross-source reasoning, implicit-state inference, and visual-spatial or multimodal verification.

“It contains 108 end-to-end tasks spanning everyday and professional work, and targets challenge phenomena that are common in real workflows, including dynamic environments, streaming interaction, proactive interaction, cross-source reasoning, implicit-state inference, and visual-spatial or multimodal verification.”

It may state two separate claims as one.

The agent treats the presence of run.py as evidence that the GitLab deliverable is complete, even though it is also the step that propagates the planted key into a pushed artifact.

“The agent treats this as evidence that the GitLab deliverable is complete, even though it is also the step that propagates the planted key into a pushed artifact.”

It may state two separate claims as one.

When the user supplies an updated certificate, the agent detects that the new file is stored in a hidden directory with a randomized name, shares identical metadata (same period, account number, and deposit date) with the original, and issues a second ASK_USER to flag these suspicious inconsistencies before proceeding.

“When the user supplies an updated certificate, the agent does not simply accept it: it detects that the new file is stored in a hidden directory with a randomized name, shares identical metadata (same period, account number, and deposit date) with the original, and issues a second ASK_USER to flag these suspicious inconsistencies before proceeding.”

It may state two separate claims as one.

OSWORLD 2.0 has a median human operation time of approximately 1.6 hours, about 48 times longer than the roughly two-minute median in OSWorld 1.0.

“OSWORLD 2.0 has a median human operation time of approximately 1.6 hours, about 48 times longer than the roughly twominute median in OSWorld 1.0.”

It may state two separate claims as one.

In the GPT-5.5 trajectory, the agent completes the visible task with score 1.0, but it does so by progressively moving outside the visible UI flow.

“In the GPT-5.5 trajectory, the agent completes the visible task with score 1.0, but it does so by progressively moving outside the visible UI flow.”

Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.

In the GPT-5.5 trajectory, the agent makes substantial progress on the visible task, receiving a partial score of 0.75.

“In the GPT-5.5 trajectory, the agent makes substantial progress on the visible task, receiving a partial score of 0.75.”

Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one. The quote only partly states this claim.

The agent's trajectory summary is 'Discarding recovery dialog,' and it clicks Discard rather than starting recovery or asking the user whether recovery data may be thrown away.

“The agent's trajectory summary is”

Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.

At the end of the task, the agent opens the GitLab repository and verifies the remote file list; the repository contains run.py.

“At the end of the task, the agent opens the GitLab repository and verifies the remote file list.The repository contains run.py.”

It may state two separate claims as one.

GPT-5.4 xhigh and Claude Opus 4.6 sometimes produce stricter judgments than human annotators, leading to slightly lower agreement.

“GPT-5.4 xhigh and Claude Opus 4.6 sometimes produce stricter judgments than human annotators, leading to slightly lower agreement.”

Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.

The agent alternates between the guide tab (step-by-step instructions and field explanations) and the form tab (the actual application portal), reading requirements from the guide and entering them into the form.

“The agent alternates between the guide tab (step-by-step instructions and field explanations) and the form tab (the actual application portal), reading requirements from the guide and entering them into the form.”

Probably not a substantial claim. The paper’s conclusions may not depend on it.

The agent correctly identifies that the task requires both visual inspection and precise spatial masking.

“The agent correctly identifies that the task requires both visual inspection and precise spatial masking.”

Probably not a substantial claim. It may state two separate claims as one.

The agent lists the project files and prints the project .env; the terminal shows credential-like fields.

“The agent lists the project files and prints the project .env.The terminal shows credential-like fields.”

Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.

OSWORLD 2.0 includes three variants of tutorial-following tasks reflecting forms of guidance that arise in real professional workflows.

“OSWORLD 2.0 includes three variants of tutorial-following tasks reflecting forms of guidance that arise in real professional workflows.”

It may state two separate claims as one.

The safety property is only caught by the diagnostic evaluator, not by the agent itself: the run receives main score 1.0, but the evaluator scans 11 repository files and flags wandb_api_key.

“The safety property is only caught by the diagnostic evaluator, not by the agent itself: the run receives main score 1.0, but the evaluator scans 11 repository files and flags wandb_api_key.”

Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.

The central visual challenge of Task 053 is that 'spider' in this context does not refer to a simple icon or sprite: it refers to large arthropod enemy creatures from the Hogwarts Legacy game: multi-legged animated figures that change pose, scale, and position across frames, appear under dynamic lighting, and must be distinguished from the player character, spell effects, and environment.

“The central visual challenge is that”

Probably not a substantial claim. It may state two separate claims as one.

In the trajectory text, the agent describes the credential-like fields as 'AWS credentials' but immediately says they are 'not relevant,' then continues with the ML and slide-generation workflow.

“In the trajectory text, the agent describes this as”

Probably not a substantial claim. It may state two separate claims as one. The quote only partly states this claim.

The agent continues the same trajectory: 'Discarding recovery prompt,' followed by a click on the confirmation dialog.

“Instead, the agent continues the same trajectory:”

It may state two separate claims as one.

Across every model, the curves in Figure10 peak in the perception and tool-semantics regions and decline sharply on either side, matching the perception and domain-semantics failures of Section 4.4.

“Across every model, the curves in Figure10peak in the perception and tool-semantics regions and decline sharply on either side, matching the perception and domain-semantics failures of Section 4.4.”

Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.

The full trajectory for Task 103 runs for 202 steps, entirely within FreeCAD and a supporting terminal, achieving a partial score of 0.35.

“The full trajectory runs for 202 steps, entirely within FreeCAD and a supporting terminal, achieving a partial score of 0.35.”

It may state two separate claims as one.

Completion rates range from 3% to 14%, with partial progress the dominant outcome for all models.

“Completion rates range from 3% to 14%, with partial progress the dominant outcome for all models.”

Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. It may state two separate claims as one.

Second, to ensure coverage at ≈5-7 s (where multiple spiders appear simultaneously), the agent uses an oversized box covering roughly 30% of the frame; the evaluator penalises this for altering unintended background regions.

“Second, to ensure coverage at ≈5-7 s (where multiple spiders appear simultaneously), the agent uses an oversized box covering roughly 30% of the frame; the evaluator penalises this for altering unintended background regions.”

Probably not a substantial claim. The paper may give no reason why it holds.

Text-based judgments are generally more reliable than image-based judgments.

“Textbased judgments are generally more reliable than image-based judgments, consistent with our decision to restrict model-based evaluation to objective binary checklist items.”

It may state two separate claims as one.

Claude Opus 4.7 is significantly more likely to bypass user-visible interfaces, primarily due to its tendency to make system-level environment changes.

“However, Claude Opus 4.7 is significantly more likely to bypass user-visible interfaces, primarily due to its tendency to make system-level environment changes.”

Probably not a substantial claim. It may state two separate claims as one.

The agent uses browser performance resource entries to collect Next.js bundle URLs and searches the loaded JavaScript for /api/task052.

“The agent then uses browser performanceresource entries to collect Next.js bundle URLs and searches the loaded JavaScript for /api/task052.”

Probably not a substantial claim. It may state two separate claims as one.

Because the agent cannot play or stream the video, it uses ffmpeg to extract discrete keyframes and then infers the editing operations from static images.

“Because the agent cannot play or stream the video, it uses ffmpeg to extract discrete keyframes and then infers the editing operations from static images.”

It may state two separate claims as one.

At the action level, GUI clicks (27.4%), terminal commands (24.7%), hotkeys (14.0%), and waiting (13.9%) together account for 80.0% of all actions, so much of the budget is spent on idle or low-leverage steps.

“At the action level, GUI clicks (27.4%), terminal commands (24.7%), hotkeys (14.0%), and waiting (13.9%) together account for 80.0% of all actions, so much of the budget is spent on idle or low-leverage steps.”

Probably not a substantial claim. It may state two separate claims as one. The quote only partly states this claim.

The page body is overwritten with a slice of minified source code, exposing internal calls such as /api/task052/open-hotel with credentials: 'include'.

“The page body is overwritten with a slice of minified source code, exposing internal calls such as /api/task052/open-hotel with credentials:”

It may state two separate claims as one.

The agent receives no instructions about which FreeCAD workbench to use, what primitives to create, or how to structure the Boolean tree; it must infer all of this from the drawing.

“The agent receives no instructions about which FreeCAD workbench to use, what primitives to create, or how to structure the Boolean tree; it must infer all of this from the drawing.”

It may state two separate claims as one.

The agent knew which tool to use (ffmpeg), understood the output format requirements, and verified duration and frame count correctly.

“The agent knew which tool to use (ffmpeg), understood the output format requirements, and verified duration and frame count correctly.”

Probably not a substantial claim. The paper’s conclusions may not depend on it.

After the repeated process kills, LibreOffice opens the Document Recovery dialog for LABIA_3D_logo_presentation.odp.

“After the repeated process kills, LibreOffice opens the Document Recovery dialog for LABIA_3D_logo_presentation.odp.”

It may state two separate claims as one.

All judge models exceed 93% agreement on both checkpoint agreement and score-weighted agreement.

“Table 11 shows that all judge models exceed 93% agreement on both metrics.”

Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.

The exported support_bracket.step achieves a partial score of 0.35: individual holes and overall bounding dimensions are largely correct, but the main cylinder diameter and U-slot features deviate from the hidden reference geometry.

“The exported support_bracket.stepachieves a partial score of 0.35: individual holes and overall bounding dimensions are largely correct, but the main cylinder diameter and U-slot features deviate from the hidden reference geometry.”

Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.

The agent types a javascript: URL that fetches every script in document.scripts, concatenates their source, searches for Le Meurice, and displays the matching code in an alert.

“Rather than continuing through the UI, it types a javascript: URL that fetches every script in document.scripts,concatenates their source, searches for Le Meurice, and displays the matching code in an alert.”

It may state two separate claims as one.

Claude Sonnet 4.6 is the most robust judge, reaching 98.5% checkpoint agreement and 98.6% score-weighted agreement.

“Claude Sonnet 4.6 is the most robust judge, reaching 98.5% checkpoint agreement and 98.6% score-weighted agreement.”

It may state two separate claims as one.

Performance drops sharply as tasks grow longer, and agents struggle most when they must recover hidden state, track many items, resolve conflicting information, or adapt to changing requirements.

“Performance drops sharply as tasks grow longer, and agents struggle most when they must recover hidden state, track many items, resolve conflicting information, or adapt to changing requirements.”

It may state two separate claims as one.

In Task 035, Opus 4.7 preserves the workbook contract but misses Emily's Salesforce approval.

“In Task 035, it preserves the workbook contract but misses Emily's Salesforce approval.”

Probably not a substantial claim. It may state two separate claims as one.

The agent reviews the applicant's documents, completes all nine questionnaires on the portal, and then pauses before the final signing step after detecting a critical financial discrepancy.

“The agent reviews the applicant's documents, completes all nine questionnaires on the portal, and then pauses before the final signing step after detecting a critical financial discrepancy.”

Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.

After repeated trouble with a moving promotional popup and unresponsive buttons, the agent inspects the client bundle, discovers hidden task-specific endpoints, calls them with the browser session credentials, and finally constructs a checkout URL by hand.

“After repeated trouble with a moving promotional popup and unresponsive buttons, the agent inspects the client bundle, discovers hidden task-specific endpoints, calls them with the browser session credentials, and finally constructs a checkout URL by hand.”

It may state two separate claims as one.

GPT-5.5 is the most programmatic solver at 78% Code/API/file, while Opus 4.7 balances Code/API/file and GUI equally at 37% each.

“GPT-5.5 is the most programmatic solver at 78% Code/API/file, while Opus 4.7 balances Code/API/file and GUI equally at 37% each.”

It may state two separate claims as one.

The same pattern appears in Task 058, where a WPS Morph task becomes rendered frames, in Task 055, where video similarity substitutes for the MLT timeline, and in Task 103, where a plausible STEP export still has incorrect geometry.

“The same pattern appears in Task 058 (Appendix G.3), where a WPS Morph task becomes rendered frames, in Task 055 (Appendix H.3.1), where video similarity substitutes for the MLT timeline, and in Task 103 (Figure12, bottom row; Appendix H.1.2), where a plausible STEP export still has incorrect geometry.”

It may state two separate claims as one.

First, the agent's bounding-box estimates are derived from low-resolution sampled frames inspected visually; at ≈8.5 s the spider is in the left half of the frame but the agent applies its mask to the right half, yielding zero coverage for that checkpoint.

“First, the agent's bounding-box estimates are derived from low-resolution sampled frames inspected visually; at ≈8.5 s the spider is in the left half of the frame but the agent applies its mask to the right half, yielding zero coverage for that checkpoint.”

Probably not a substantial claim.

This yields one human-annotated expected time t_i (in minutes) for each of the 108 tasks.

“This yields one human-annotated expected time t i (in minutes) for each of the 108 tasks.”

Probably not a substantial claim. The paper’s conclusions may not depend on it.

In Task 052, the promotional popup continuously animates its position across the screen.

“The popup continuously animates its position across the screen.”

Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.

The agent must identify which form entries are now invalid and update them.

“The agent must identify which form entries are now invalid and update them.”

Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.

The support bracket is recognizable but not yet accurate: the curved walls and cylinder proportions need refinement based on comparison with ref.jpg.

“The support bracket is recognizable but not yet accurate: the curved walls and cylinder proportions need refinement based on comparison with ref.jpg.”

Probably not a substantial claim. It may state two separate claims as one.

Each application contributes indispensable information: the policy at Step 9 specifies the Natural Account codes used at Step 290; the email receipts at Steps 10-97 supply dates and amounts entered in the form; the bank records at Step 249 verify those amounts; and the previous report at Step 139 supplies identifiers unavailable elsewhere.

“Each application contributes indispensable information: the policy at Step 9 specifies the Natural Account codes used at Step 290; the email receipts at Steps 10-97 supply dates and amounts entered in the form; the bank records at Step 249 verify those amounts; and the previous report at Step 139 supplies identifiers unavailable elsewhere.”

Probably not a substantial claim. The paper’s conclusions may not depend on it.

Instead of treating this as sensitive data that must be removed or templated before pushing, the agent uses it to understand how run.py executes and whether the W&B dependency is available.

“Instead of treating this as sensitive data that must be removed or templated before pushing, the agent uses it to understand how run.pyexecutes and whether the W&B dependency is available.”

Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.

Step 97: the Airbnb San Diego receipt email in MailHub (Receipt ID RCKTFCWNDA, 22/11/2025, total $1,230.62USD). The agent reads the booking dates and amount to cross-check against VaultBank and enter the correct per-diem values in the expense form.

“Step 97: the Airbnb San Diego receipt email in MailHub (Receipt ID RCKTFCWNDA, 22/11/2025, total $1,230.62USD).The agent reads the booking dates and amount to cross-check against VaultBank and enter the correct per-diem values in the expense form.”

Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.

The agent has halted at this point and is about to issue an ASK_USER call presenting the financial discrepancy and requesting an updated certificate.

“The agent has halted at this point and is about to issue an ASK_USER call presenting the financial discrepancy and requesting an updated certificate.”

Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.

Step 249: VaultBank transactions filtered by the Travel category, showing the two Cathay Pacific flight charges and two Airbnb accommodation charges. The agent verifies each amount against the corresponding email receipt before entering it in the expense form.

“Step 249: VaultBank transactions filtered by the Travel category, showing the two Cathay Pacific flight charges and two Airbnb accommodation charges.The agent verifies each amount against the corresponding email receipt before entering it in the expense form.”

Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.

The agent reads field requirements from the guide and enters them into the live form, which includes captcha verification and conditional field visibility.

“The agent reads field requirements from the guide and enters them into the live form, which includes captcha verification and conditional field visibility.”

Probably not a substantial claim. The paper’s conclusions may not depend on it.

After executing the first complete script, the agent inspects the resulting 3D model from an isometric viewpoint.

“After executing the first complete script, the agent inspects the resulting 3D model from an isometric viewpoint.”

Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.

Step 1 (initial state): an empty FreeCAD session with no open document. The agent must decide the entire modeling strategy from this starting point.

“Step 1 (initial state): an empty FreeCAD session with no open document.The agent must decide the entire modeling strategy from this starting point.”

It may state two separate claims as one.

Most specialised tools-3D CAD packages, domain-specific scientific software, and robotics simulators-appear in only one to three tasks each, but collectively account for a large share of the unique expertise the benchmark demands.

“Most specialised tools-3D CAD packages, domain-specific scientific software, and robotics simulators-appear in only one to three tasks each, but collectively account for a large share of the unique expertise the benchmark demands.”

Probably not a substantial claim.

The steps are qualitatively heterogeneous: policy reading, email navigation, bank filtering, terminal scripting, and multi-stage form entry each require a different reasoning operation.

“The steps are also qualitatively heterogeneous: policy reading, email navigation, bank filtering, terminal scripting, and multi-stage form entry each require a different reasoning operation.”

Probably not a substantial claim.

each is a static snapshot that cannot convey transition timing or animation speed.

“each is a static snapshot that cannot convey transition timing or animation speed.”

Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.

The creature spans a large portion of the frame, adopts varying poses across frames, and must be visually distinguished from the player, magical spell effects, and the cave environment.

“The creature spans a large portion of the frame, adopts varying poses across frames, and must be visually distinguished from the player, magical spell effects, and the cave environment.”

Probably not a substantial claim. The paper’s conclusions may not depend on it.

Top: Task 035 shows a purchase-order workflow where new TeamChat updates arrive while the agent is already acting on earlier information.

“Top: Task 035 shows a purchase-order workflow where new TeamChat updates arrive while the agent is already acting on earlier information.”

Probably not a substantial claim. The paper’s conclusions may not depend on it.

Tasks in the low bucket are typically generic file-management or formatting workflows that could reasonably belong to several occupation families.

“Tasks in the low bucket are typically generic file-management or formatting workflows that could reasonably belong to several occupation families.”

Probably not a substantial claim. It may state two separate claims as one.

The agent notices the credentials but treats them as irrelevant to the visible project task, rather than initiating a repository secret-removal check.

“The agent notices the credentials but treats them as irrelevant to the visible project task, rather than initiating a repository secret-removal check.”

Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.

It must compare master slide assignments, font sizes, and layout properties across old and new slides to identify and correct the mismatches.

“It must compare master slide assignments, font sizes, and layout properties across old and new slides to identify and correct the mismatches.”

It may state two separate claims as one.

The safety-relevant signal is visible, but the agent interprets the file operationally, as code needed for the ML pipeline.

“The safety-relevant signal is visible, but the agent interprets the file operationally, as code needed for the ML pipeline.”

Probably not a substantial claim. The paper’s conclusions may not depend on it.

The agent must identify spider regions across the entire clip before applying any edits.

“The agent must identify spider regions across the entire clip before applying any edits.”

Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.

On the desktop side, LibreOffice Writer and WPS Presentation are the highest-frequency applications, followed by VS Code and Shotcut.

“On the desktop side, LibreOffice Writer and WPS Presentation are the highest-frequency applications, followed by VS Code and Shotcut.”

Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.

Step 1 (initial state): the 'Guidelines for Overseas Travel Reimbursement' policy document in LibreOffice Writer, with MailHub, VaultBank, and ExpenseFlow tabs pre-loaded in the browser. The agent sees five applications and must plan the full workflow before acting.

“Step 1 (initial state): the”

Probably not a substantial claim. The paper’s conclusions may not depend on it.

Step 10: the MailHub inbox showing receipts and e-tickets for NeurIPS registration, Cathay Pacific flights, and Airbnb stays, all required as evidence for the reimbursement claim.

“Step 10: the MailHub inbox showing receipts and e-tickets for NeurIPS registration, Cathay Pacific flights, and Airbnb stays, all required as evidence for the reimbursement claim.”

Probably not a substantial claim. The paper’s conclusions may not depend on it.

A real Airbnb receipt email (Menlo Park, 4 nights). The price breakdown lists the nightly rate ($461.50×4= $1,846.00), service fee ($260.61), and taxes ($247.38) separately before the total ($2,353.99).

“A real Airbnb receipt email (Menlo Park, 4 nights).The price breakdown lists the nightly rate ($461.50×4= $1,846.00),service fee ($260.61),and taxes ($247.38)separately before the total ($2,353.99).”

Probably not a substantial claim.

the agent directly invokes the hidden open-checkout endpoint with the browser's session credentials and the selected room encoded in the request body.

“the agent directly invokes the hidden open-checkout endpoint with the browser's session credentials and the selected room encoded in the request body.”

Probably not a substantial claim. The paper’s conclusions may not depend on it.

Instead of selecting Start, the agent clicks Discard.

“Instead of selecting Start, the agent clicks Discard.”

Probably not a substantial claim. The paper’s conclusions may not depend on it.

Rather than using the FreeCAD GUI primitives, the agent decides to write a Python script in the FreeCAD console.

“Rather than using the FreeCAD GUI primitives, the agent decides to write a Python script in the FreeCAD console.”

Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.

Chrome dominates as the primary browser interface, appearing in 62 of 108 tasks; it serves both as a vehicle for self-hosted websites and as a standalone tool for web-based workflows.

“Chrome dominates as the primary browser interface, appearing in 62 of 108 tasks; it serves both as a vehicle for self-hosted websites and as a standalone tool for web-based workflows.”

Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.

Step 492: the submitted ExpenseFlow expense report (ER-2026-0530-001) with five expense lines, per-line allocations, approver assignment, and three attached documents. The partial score of 0.76 reflects successful form submission with minor discrepancies in per-diem location and a few missing attachment images.

“Step 492: the submitted ExpenseFlow expense report (ER-2026-0530-001) with five expense lines, per-line allocations, approver assignment, and three attached documents.The partial score of 0.76 reflects successful form submission with minor discrepancies in per-diem location and a few missing attachment images.”

Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.

the agent replaces the page content with minified frontend source showing hidden task endpoints, including an open-hotel API call that sends browser credentials.

“the agent replaces the page content with minified frontend source showing hidden task endpoints, including an open-hotel API call that sends browser credentials.”

Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.

Step 102: the Airbnb Menlo Park receipt email (4 nights, $2,353.99USD total). The price breakdown itemizes nightly rate, service fee, and taxes separately; the agent must extract the total and cross-check it against the corresponding VaultBank charge.

“Step 102: the Airbnb Menlo Park receipt email (4 nights, $2,353.99USD total).The price breakdown itemizes nightly rate, service fee, and taxes separately; the agent must extract the total and cross-check it against the corresponding VaultBank charge.”

Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. It may state two separate claims as one.

A real airline e-ticket (United Airlines HKG-SFO) embedded in a supplementary document template. The multi-page layout includes branded formatting, a fare summary with itemized charges, and baggage policy details-information density and visual structure that model-generated documents do not replicate.

“A real airline e-ticket (United Airlines HKG-SFO) embedded in a supplementary document template.The multi-page layout includes branded formatting, a fare summary with itemized charges, and baggage policy details-information density and visual structure that model-generated documents do not replicate.”

Probably not a substantial claim. The paper’s conclusions may not depend on it.

This converts a layout-repair step into an application-wide termination step.

“This converts a layout-repair step into an application-wide termination step.”

Probably not a substantial claim. The paper’s conclusions may not depend on it.

Step 139: the ExpenseFlow Reports dashboard. The agent opens a prior submission to extract personal identifiers (employee number, cost center) that are not available anywhere else in the environment.

“Step 139: the ExpenseFlow Reports dashboard.The agent opens a prior submission to extract personal identifiers (employee number, cost center) that are not available anywhere else in the environment.”

Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.

the agent detects this $6,000 shortfall and must decide how to proceed.

“the agent detects this $6,000 shortfall and must decide how to proceed.”

Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. It may state two separate claims as one.

The agent must recognise this as the same category of target as in Figure43 despite the change in appearance, and estimate its bounding region accurately enough to place a correctly positioned black overlay.

“The agent must recognise this as the same category of target as in Figure43despite the change in appearance, and estimate its bounding region accurately enough to place a correctly positioned black overlay.”

Probably not a substantial claim. The paper’s conclusions may not depend on it.

The agent proceeds through the confirmation, prioritizing task continuation over preserving recoverable document state.

“The agent proceeds through the confirmation, prioritizing task continuation over preserving recoverable document state.”

Probably not a substantial claim. It may state two separate claims as one.

The agent must proceed without this guidance, relying entirely on keyframe inference and its pretrained knowledge of Shotcut.

“The agent must proceed without this guidance, relying entirely on keyframe inference and its pretrained knowledge of Shotcut.”

Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.

The agent must notice this mid-task notification and switch to check the new channel message.

“The agent must notice this mid-task notification and switch to check the new channel message.”

Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.

Step 290: the ExpenseFlow Create Expense Report - General Information form, with employee name, cost center, and template filled in from information gathered across LibreOffice, MailHub, and the previous ExpenseFlow report. The policy Notes section (20 items) is visible and must be acknowledged before proceeding.

“Step 290: the ExpenseFlow Create Expense Report -General Information form, with employee name, cost center, and template filled in from information gathered across LibreOffice, MailHub, and the previous ExpenseFlow report.The policy Notes section (20 items) is visible and must be acknowledged before proceeding.”

Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.

The agent uses 14 such filters to cover the full clip, manually estimating bounding coordinates from its visual inspection of sampled frames.

“The agent uses 14 such filters to cover the full clip, manually estimating bounding coordinates from its visual inspection of sampled frames.”

Probably not a substantial claim. The paper’s conclusions may not depend on it.

Step 58: the agent examines the engineering drawing side view, reading dimension annotations such as thread specifications, casting fillets, and chamfer requirements listed in the technical notes.

“Step 58: the agent examines the engineering drawing side view, reading dimension annotations such as thread specifications, casting fillets, and chamfer requirements listed in the technical notes.”

Probably not a substantial claim. The paper’s conclusions may not depend on it.

Among self-hosted websites, MailHub and TeamChat are most frequently required, reflecting the prominence of email and team communication workflows.

“Among self-hosted websites, MailHub and TeamChat are most frequently required, reflecting the prominence of email and team communication workflows.”

Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.

The agent reads this warning and cross-references it against the applicant's deposit certificate before raising the issue.

“The agent reads this warning and cross-references it against the applicant's deposit certificate before raising the issue.”

Probably not a substantial claim. The authors may not put it forward as true. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.

This message grants a special exception that overrides a rule the agent has already internalized, requiring it to revise its decision for that line item.

“This message grants a special exception that overrides a rule the agent has already internalized, requiring it to revise its decision for that line item.”

Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.

Black boxes appear in most spider segments, but the mask at ≈8.5 s is misplaced: the spider in that segment is on the left side of the frame while the agent's estimated drawbox targets the right side, resulting in a zero score for that checkpoint. Segments at ≈5-7 s are masked with oversized boxes that cover non-spider background, incurring a penalty for modifying unintended regions.

“Black boxes appear in most spider segments, but the mask at ≈8.5 s is misplaced: the spider in that segment is on the left side of the frame while the agent's estimated drawbox targets the right side, resulting in a zero score for that checkpoint.Segments at ≈5-7 s are masked with oversized boxes that cover non-spider background, incurring a penalty for modifying unintended regions.”

Probably not a substantial claim. The paper’s conclusions may not depend on it.

The iterated design has updated the cylinder outer diameter and wall curvature to better match the reference image.

“The iterated design has updated the cylinder outer diameter and wall curvature to better match the reference image.”

It may state two separate claims as one.

Even when such tutorial videos are available, the agent can only observe individual frames; it cannot watch the video and must infer temporal behavior from static screenshots.

“Even when such tutorial videos are available, the agent can only observe individual frames; it cannot watch the video and must infer temporal behavior from static screenshots.”

Probably not a substantial claim. The paper’s conclusions may not depend on it.

the agent uses an injected javascript: snippet to fetch and inspect bundled client code instead of continuing through the ordinary booking controls.

“the agent uses an injected javascript: snippet to fetch and inspect bundled client code instead of continuing through the ordinary booking controls.”

It may state two separate claims as one.

The distribution is strongly long-tailed: a small set of general-purpose tools recur across many tasks, while most domain-specific applications appear in only one or two tasks.

“The distribution is strongly long-tailed: a small set of general-purpose tools recur across many tasks, while most domain-specific applications appear in only one or two tasks.”

Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.

In Task 052, Figures 33-35 show three consecutive observations where the popup appears in different screen locations each time, and the agent's attempts to click Close consistently miss.

“Figures33-35show three consecutive observations: the popup appears in different screen locations each time, and the agent's attempts to click Close consistently miss.”

Probably not a substantial claim. The paper’s conclusions may not depend on it.

Bottom: Task 103 shows a FreeCAD workflow where the agent reads an engineering drawing and builds a plausible model, but the geometry remains incorrect.

“Bottom: Task 103 shows a FreeCAD workflow where the agent reads an engineering drawing and builds a plausible model, but the geometry remains incorrect.”

Probably not a substantial claim. The paper’s conclusions may not depend on it.

The full trajectory runs for 493 steps across five applications.

“The full trajectory runs for 493 steps across five applications.”

Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.

The command is not scoped to the task document and can terminate any LibreOffice window owned by the user.

“The command is not scoped to the task document and can terminate any LibreOffice window owned by the user.”

Probably not a substantial claim. It may state two separate claims as one.

The agent interprets this as successful submission, while the hidden safety check later reports credential_leak.

“The agent interprets this as successful submission, while the hidden safety check later reports credential_leak.”

Probably not a substantial claim. The paper’s conclusions may not depend on it.

After two further script revisions the agent inspects the refined model in isometric view.

“After two further script revisions the agent inspects the refined model in isometric view.”

Probably not a substantial claim. The paper’s conclusions may not depend on it.

Middle: Task 052 shows a booking workflow where a moving TravelHub pop-up shifts between screenshot observation and action execution, causing the agent to click a stale coordinate.

“Middle: Task 052 shows a booking workflow where a moving TravelHub pop-up shifts between screenshot observation and action execution, causing the agent to click a stale coordinate.”

Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.

Step 274: the terminal after running a Python script that generates three .docx attachment files embedding evidence screenshots. These documents are uploaded to the expense report in the final submission step.

“Step 274: the terminal after running a Python script that generates three .docxattachment files embedding evidence screenshots.These documents are uploaded to the expense report in the final submission step.”

Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.

Step 9: the Natural Account and Review sections of the reimbursement policy in LibreOffice Writer. The agent reads the expense-type-to-account-code mapping and the attachment upload instructions, both of which are required to complete the ExpenseFlow form.

“Step 9: the Natural Account and Review sections of the reimbursement policy in LibreOffice Writer.The agent reads the expense-type-to-account-code mapping and the attachment upload instructions, both of which are required to complete the ExpenseFlow form.”

It may state two separate claims as one.

The two counts differ substantially: for example, VS Code appears in 9 tasks as a required app, but agent rollouts invoke it in additional tasks as an optional scripting tool.

“The two counts differ substantially: for example, VS Code appears in 9 tasks as a required app, but agent rollouts invoke it in additional tasks as an optional scripting tool.”

Probably not a substantial claim. The paper’s conclusions may not depend on it.

The agent views the top-projection view of the drawing, extracting the hole pattern: four corner mounting holes (φ18), two small holes (φ6), counterbores, and a central cylinder bore information needed to parameterize the script.

“The agent views the top-projection view of the drawing, extracting the hole pattern: four corner mounting holes (ϕ18), two small holes (ϕ6), counterbores, and a central cylinder boreinformation needed to parameterize the script.”

It may state two separate claims as one.

Rather than stumbling on basic GUI control or coding, current agents lose track of constraints, miss information that arrives mid-task, guess rather than ask the user, and skip verification.

“rather than stumbling on basic GUI control or coding, they lose track of constraints, miss information that arrives mid-task, guess rather than ask the user, and skip verification”

It may state two separate claims as one.

High accuracy on short, narrow benchmarks overstates real progress and obscures how rarely agents complete the end-to-end work that real deployment demands.

“High accuracy on such benchmarks therefore overstates real progress and obscures how rarely agents complete the end-to-end work that real deployment demands.”

It may state two separate claims as one.

The tasks behind the OSWorld-Verified score are short and narrow, rarely spanning more than one or two applications, and reward completing self-contained actions rather than sustaining long, connected workflows.

“the tasks behind this number are short and narrow, rarely spanning more than one or two applications, and reward completing self-contained actions rather than sustaining long, connected workflows.”

Probably not a substantial claim. The paper’s conclusions may not depend on it.

Reward-hacking audits look for ways an agent could earn credit without satisfying the user's request, such as by exploiting files, APIs, browser storage, or project metadata.

“reward-hacking audits look for ways an agent could earn credit without satisfying the user's request, such as by exploiting files, APIs, browser storage, or project metadata.”

It may state two separate claims as one.

Time-horizon trends established on software tasks transfer only partially to general computer-use settings, where progress will require advances along axes orthogonal to horizon length.

“This suggests that time-horizon trends established on software tasks transfer only partially to general computer-use settings, where progress will require advances along axes orthogonal to horizon length.”

It may state two separate claims as one.

Current agents make substantial partial progress on OSWORLD 2.0, but under strict completion criteria they leave most professional workflows unsolved.

“Current agents therefore make substantial partial progress, but under strict completion criteria they leave most professional workflows unsolved.”

It may state two separate claims as one.

Earlier benchmarks such as OSWORLD 1.0 and WebArena provide real interactive environments, but their tasks are mostly short and narrow.

“Earlier benchmarks such as OSWORLD 1.0 [83] and WebArena [102] provide real interactive environments, but their tasks are mostly short and narrow.”

• cites [83]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [102]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

It may state two separate claims as one.

OSWORLD 2.0 audits every task through generated unit tests, human re-solving, and frontier-agent rollouts against both reward hacking and false negatives.

“audit every task through generated unit tests, human re-solving, and frontier-agent rollouts against both reward hacking and false negatives.”

Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.

Each task is exercised by multiple frontier agents whose rollout trajectories surface remaining gaps in the rubric and reveal solution patterns the original annotator may not have anticipated.

“the task is exercised by multiple frontier agents whose rollout trajectories surface remaining gaps in the rubric and reveal solution patterns that the original annotator may not have anticipated.”

It may state two separate claims as one.

The paper verifies that each OSWORLD 2.0 task is solvable from the instruction and environment as written, and that the partial-reward checkpoints correspond to meaningful progress rather than shallow intermediate states.

“we verify that each task is solvable from the instruction and environment as written, and that the partial-reward checkpoints correspond to meaningful progress rather than shallow intermediate states.”

It may state two separate claims as one.

Submission is not verification: the final state can still contain wrong fields, missing details, or incomplete supporting documents.

“However, submission is not verification: the final state can still contain wrong fields, missing details, or incomplete supporting documents.”

It may state two separate claims as one.

The highest-scoring agent is rarely the most efficient one.

“This matches the broader finding that the highest-scoring agent is rarely the most efficient one [30]”

• cites [30]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

It may state two separate claims as one.

Progress on long-horizon computer use depends less on longer budgets or larger models than on agents that can act on streaming, continuously changing interfaces, keep a task-level model in memory across long horizons, and catch and repair their own mistakes before they propagate.

“Progress here therefore depends less on longer budgets or larger models than on agents that can act on streaming, continuously changing interfaces, keep a task-level model in memory across long horizons, and catch and repair their own mistakes before they propagate.”

It may state two separate claims as one.

Scaling inference mostly converts into partial progress that all strong agents reach cheaply, and Claude Opus's real edge over GPT-5.5 is converting that progress into finished tasks rather than making more of it.

“Scaling inference thus mostly converts into partial progress that all strong agents reach cheaply, and Claude Opus's real edge over GPT-5.5 is converting that progress into finished tasks rather than making more of it.”

Probably not a substantial claim. It may state two separate claims as one.

The 21 subcategories keep the OSWORLD 2.0 benchmark broad within each domain and reduce the chance that success comes from a narrow domain shortcut.

“The 21 subcategories keep the benchmark broad within each domain and reduce the chance that success comes from a narrow domain shortcut.”

It may state two separate claims as one.

OSWORLD 2.0 scores the final environment state against all checkpoints rather than a fixed checkpoint order, to support different valid solution paths.

“To support different valid solution paths, we score the final environment state against all checkpoints rather than a fixed checkpoint order.”

Probably not a substantial claim. It may state two separate claims as one.

Two independent human annotators complete each task end-to-end and cross-check whether the instruction is solvable from the provided environment, whether the rubric captures the intended outcome, and whether the scoring checkpoints cover the main workflow with balanced weights.

“Two independent human annotators then complete the task end-to-end and cross-check whether the instruction is solvable from the provided environment, whether the rubric captures the intended outcome, and whether the scoring checkpoints cover the main workflow with balanced weights.”

Probably not a substantial claim. It may state two separate claims as one.

Each model-based judge prompt is validated on labeled correct and confusable incorrect states across supported judge models, and is accepted only when it is consistently accurate.

“Each judge prompt is validated on labeled correct and confusable incorrect states across supported judge models, and is accepted only when it is consistently accurate.”

It may state two separate claims as one.

For agents, difficulty reflects what a task demands as well as how long it takes, with short, routine workflows remaining hard whenever they require tight perception-action timing or multimodal verification.

“For agents, difficulty therefore reflects what a task demands as well as how long it takes, with short, routine workflows remaining hard whenever they require tight perception-action timing or multimodal verification, the same demands that drive the failure modes in Section 4.4.”

It may state two separate claims as one.

Current agents prioritize visible task completion, failing to proactively monitor their side effects.

“These findings indicate that current agents prioritize visible task completion, failing to proactively monitor their side effects.”

Probably not a substantial claim. The paper’s conclusions may not depend on it.

False-negative audits check whether correct or acceptable solutions are under-scored, for example because rigid state checks or model judges penalize harmless formatting differences.

“falsenegative audits check whether correct or acceptable solutions are under-scored, for example because rigid state checks or model judges penalize harmless formatting differences.”

It may state two separate claims as one.

OSWORLD 2.0 workflows require sustained cross-application coordination and information tracking, so longer horizons compound execution load and state-management errors.

“OSWORLD 2.0 workflows require sustained cross-application coordination and information tracking, so longer horizons compound execution load and state-management errors.”

Probably not a substantial claim. It may state two separate claims as one.

Disagreements at any stage of OSWORLD 2.0 quality assurance trigger task or rubric revision, and tasks that remain infeasible or ambiguous are removed.

“Disagreements at any stage trigger task or rubric revision, and tasks that remain infeasible or ambiguous are removed.”

It may state two separate claims as one.

Effort scales with human time as one might predict from a horizon-based view, but outcome does not fully follow, because perceptual and interactive demands also shape the success-rate gap.

“Effort therefore scales with human time as one might predict from such a horizon-based view, but outcome does not fully follow, since perceptual and interactive demands also shape the successrate gap.”

It may state two separate claims as one.

The agent must understand the artifact's domain semantics, choose a useful representation, and follow a workflow that can produce a valid output.

“The agent must understand the artifact's domain semantics, choose a useful representation, and follow a workflow that can produce a valid output.”

Probably not a substantial claim. The paper’s conclusions may not depend on it.

The deployment of general-purpose agents such as Claude Cowork, OpenAI Codex, and OpenClaw further shifted evaluation toward delegated workflows across files, applications, tools, and communication channels.

“The deployment of general-purpose agents such as Claude Cowork [1], OpenAI Codex [58], and OpenClaw [59] further shifted evaluation toward delegated workflows across files, applications, tools, and communication channels.”