OS-Themis: A Scalable Critic Framework for Generalist GUI Rewards

May combine two claims.

Native GUI agents demonstrate proficiency in digital navigation but suffer from poor error recovery in stochastic environments, often failing when deviating from expert trajectories.

“they suffer from poor error recovery in stochastic environments, often failing when deviating from expert trajectories”

• cites Qin et al., 2025 sourcePartly supported

  Section 4.5 (p. 20) says agents can get stuck without error-correction capabilities and that training on idealized, error-free trajectories leaves them unable to learn recovery. It does not specifically establish frequent failure upon deviation from expert trajectories in stochastic environments; it also describes reflection tuning to improve UI-TARS’s recovery. (LLM, conf=0.84)

  Experimental — accuracy not yet measured
• cites Liu et al., 2025 sourcePartly supported

  Figure 6 (p. 26) shows ScaleCUA repeatedly taking an ineffective action after missing a GUI element, supporting a concern about error recovery. But A.5.6 describes stochasticity in rule-driven data-collection agents, not stochastic environments in which native GUI agents fail; the cited passages do not link failures to deviations from expert trajectories or establish the claim for native GUI agents generally. (LLM, conf=0.86)

  Experimental — accuracy not yet measured
• cites Lu et al., 2025Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

Compared with ZeroGUI and DigiRL, OS-Themis exhibits consistently strong performance with superior stability and overall effectiveness across diverse platforms.

“compared with ZeroGUI (Yang et al., 2025) and DigiRL (Bai et al., 2024), OS-Themis exhibits consistently strong performance with superior stability and overall effectiveness across diverse platforms.”

• cites Yang et al., 2025 sourceSupported
• cites Bai et al., 2024 sourceSupported

May combine two claims.

Some existing training-based GUI critic models are PRM-style models and are not compatible with OGRBench evaluation.

“some existing critics are PRM-style models and are not compatible with OGRBench evaluation (Wu et al., 2025b;Wanyan et al., 2025;Gu et al., 2025).”

• cites Wu et al., 2025bNot matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Wanyan et al., 2025 sourcePartly supported

  Section 3.3 identifies GUI-Critic-R1 as a trained GUI pre-critic model, supporting its use as an example of a training-based critic. The paper does not describe it as PRM-style or address compatibility with OGRBench; searches for “PRM-style,” “process reward model,” and “OGRBench” found no supporting passage. No page numbers were available in our copy. (LLM, conf=0.78)

  Experimental — accuracy not yet measured
• cites Gu et al., 2025Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

Gemini-3-Flash achieves the highest precision across all models while maintaining sufficiently high recall, indicating more reliable positive judgments.

“Gemini-3-Flash (Comanici et al., 2025) achieves the highest precision across all models while maintaining sufficiently high recall, indicating more reliable positive judgments.”

• cites Comanici et al., 2025 sourceCould not verify

  The introduction and Section 3 evaluate Gemini 2.X models, and the model list in Section 3 includes no Gemini-3-Flash. Searches for Gemini-3-Flash, precision, recall, and positive judgments found no passage assessing the claimed performance; the paper's discussion of other models does not establish a contrary result. (LLM, conf=0.78)

  Experimental — accuracy not yet measured

May combine two claims.

Using the entire unfiltered collected dataset for SFT degrades model performance to varying extents, indicating substantial noise in the raw data.

“when the entire unfiltered collected dataset is used for SFT, the model performance degrades to varying extents, indicating the presence of substantial noise in the raw data.”

May combine two claims.

For Qwen3-VL-8B, fine-tuning with OS-Themis yields a 7.1% gain over the baseline on AndroidWorld, improving over ZeroGUI by 3% and SEAgent by 4.7%.

“for Qwen3-VL-8B, fine-tuning with OS-Themis yields a 7.1% gain over the baseline, while improving over ZeroGUI and SEAgent by 3% and 4.7%, respectively.”

May combine two claims.

Removing the Verifier Agent drops Accuracy by 6.1% and Precision by 15.6%.

“Accuracy drops by 6.1%, while Precision decreases even more substantially by 15.6%.”

May combine two claims.

Fine-tuning on OS-Themis-filtered data brings improvements of 6.9% and 5.0% over the respective baselines on Qwen3-VL-4B and Qwen3-VL-8B, and outperforms fine-tuning on data filtered by DigiRL and ZeroGUI.

“fine-tuning on the OS-Themis-filtered data brings substantial improvements of 6.9% and 5.0% over the respective baselines, and clearly outperforms fine-tuning on data filtered by DigiRL and ZeroGUI.”

May combine two claims.

On average, OS-Themis outperforms DigiRL by 18.8% in accuracy, 29.6% in precision, 16.9% in recall, and 26.2% in F1-score on OGRBench.

“On average, our method outperforms DigiRL by significant margins, including 18.8% in accuracy, 29.6% in precision, 16.9% in recall, and 26.2% in F1-score.”

May combine two claims.

After scaling training to 1024 tasks, Qwen3-VL-4B achieves a score of 55.6% on AndroidWorld, improving by 10.3% over the baseline.

“after scaling training to 1024 tasks, Qwen3-VL-4B achieves a score of 55.6% on AndroidWorld, improving by 10.3% over the Baseline.”

May combine two claims.

Removing the Selector Agent drops Accuracy by about 4.7% and Precision by roughly 13.1% on OGRBench.

“Accuracy drops by about 4.7%, and Precision decreases even more sharply by roughly 13.1%.”

May combine two claims.

OS-Themis is effective in driving capability gains in Online RL and iterative data filtering.

“empirical results in Online RL and iterative data filtering further underscore its effectiveness in driving capability gains.”

May combine two claims.

OS-Themis surpasses ZeroGUI by 7.7% in accuracy, 5.1% in precision, 13.0% in recall, and 13.4% in F1-score on OGRBench.

“Similarly, it surpasses ZeroGUI by 7.7%, 5.1%, 13.0%, and 13.4% across the respective metrics.”

May combine two claims.

Incorporating the Reviewer maintains overall Accuracy while substantially boosting Precision by expanding the diversity and coverage of extracted evidence.

“incorporating the Reviewer maintains overall Accuracy while substantially boosting Precision by expanding the diversity and coverage of extracted evidence.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

OS-Themis demonstrates robust initial scalability and stability.

“While OS-Themis demonstrates robust initial scalability and stability”

May combine two claims.

For the Qwen3-VL-4B backbone, employing OS-Themis leads to a 6% absolute improvement over the baseline on AndroidWorld, outperforming ZeroGUI (+5.2%) and SEAgent (+3.5%).

“For the Qwen3-VL-4B backbone, employing OS-Themis leads to a 6% absolute improvement over the baseline, significantly outperforming concurrent methods like ZeroGUI (+5.2%) and SEAgent (+3.5%).”

May combine two claims.

Across all tested base models, OS-Themis consistently yields superior results in both accuracy and precision.

“Across all tested base models, OS-Themis consistently yields superior results in both accuracy and precision.”

May combine two claims.

Incorporating assignment goals significantly mitigates the Verifier's excessive leniency and results in a marked improvement in model precision.

“Conversely, incorporating assignment goals significantly mitigates this issue, resulting in a marked improvement in model precision.”

May combine two claims.

Scaling the Judge/Verifier is more critical than scaling other components, while scaling the Reviewer mainly improves precision.

“results suggest that scaling the Judge/Verifier is more critical, while scaling the Reviewer mainly improves precision.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

Current exploration of reward shaping remains preliminary.

“our current exploration of reward shaping remains preliminary.”

May combine two claims.

GPT-5 and GPT-5-mini tend to produce more conservative outputs and prefer predicting False, which can lead to notably low recall across different frameworks.

“GPT-5 and GPT-5-mini(OpenAI, 2025) tend to produce more conservative outputs and prefer predicting False, which can lead to notably low recall across different frameworks”

• cites OpenAI, 2025Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing).

The Reviewer Agent identified three specific issues in the case study: case sensitivity discrepancies, failure to save the file, and incorrect cursor positioning.

“the Reviewer Agent, which identifies three specific issues: case sensitivity discrepancies, failure to save the file, and incorrect cursor positioning.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

The Reviewer Agent within the Verdict Calibration Module correctly identifies potential issues in the interaction information and feeds them back to the Milestone Verification Module.

“The Reviewer Agent within the Verdict Calibration Module correctly identifies potential issues in the interaction information and feeds them back to the Milestone Verification Module.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

SEAgent achieves overall accuracy 61.8, precision 89.7, recall 26.1 on OmniGUIRewardBench.

“SEAgent 59.4 94.2 24.9 55.9 85.7 18.4 63.9 87.0 21.3 84.4 100.0 25.0 65.8 82.7 43.4 61.8 89.7 26.1”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

OmniGUIRewardBench covers Ubuntu, Android, Windows, macOS, and Web, and contains 1409 samples.

“OmniGUIRewardBench | ✓ | ✓ | ✓ | ✓ | ✓ | 1409”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims. The quote only partly states this claim.

DigiRL achieves mean overall accuracy 62.8, precision 61.3, recall 53.5 across platforms in OmniGUIRewardBench.

“Mean 60.4 62.9 49.9 65.1 61.5 67.4 67.1 60.4 55.7 75.5 37.3 39.8 59.6 60.3 54.4 62.8 61.3 53.5”

May combine two claims.

OS-Themis maintains a consistently high level of Precision on AgentRewardBench, reaching a maximum of 76.8, close to the rule-based method and significantly higher than other models.

“our model maintains a consistently high level of Precision, reaching a maximum of 76.8, which is close to the rule-based method and significantly higher than that of other models.”

Probably not a substantial claim: may not pass S3 (load-bearing).

On OmniGUIRewardBench, per-trajectory average calls is 14.1.

“Calls | 14.1”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

In the example of editing a file in Markor, the agent failed the exact text requirement (case mismatch) and failed to provide evidence of saving the document.

“The agent failed the exact text requirement (case mismatch) and failed to provide evidence of saving the document.”

May combine two claims.

Majority Voting exhibits the strongest robustness among the three strategies and maintains relatively stable accuracy across different k.

“Majority Voting exhibits the strongest robustness among the three.As a balanced strategy, it maintains relatively stable accuracy across different k”

May combine two claims.

OS-Themis w/ Reviewer (Advisor) achieves Overall Acc 87.8, Prec 88.7, Rec 86.4.

“OS-Themis w/ Reviewer (Advisor) 87.8 88.7 86.4”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

CUARewardBench covers only Ubuntu and contains 272 samples.

“CUARewardBench | ✓ | ✗ | ✗ | ✗ | ✗ | 272”

May combine two claims.

In the vast majority of experiments, the framework terminates well before hitting the worst-case caps, while still guaranteeing bounded resource usage.

“Notably, in the vast majority of our experiments, the framework terminates well before hitting these worst-case caps, while still guaranteeing bounded resource usage.”

Probably not a substantial claim: may not pass S3 (load-bearing).

In the case study, the GUI Agent ultimately failed to complete the task satisfactorily.

“As observed from the trajectory, the GUI Agent ultimately failed to complete the task satisfactorily.”

May combine two claims.

The task-level average milestone percentage is 58.47%, with a median of 66.67%.

“The task-level average milestone percentage is 58.47%, with a median of 66.67%.”

May combine two claims.

Despite the tradeoff, overall accuracy can still improve marginally under Any Voting, reflecting the net benefit of reducing misses.

“Despite this tradeoff, overall accuracy can still improve marginally, reflecting the net benefit of reducing misses under this aggregation rule.”

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

During training, open-source Qwen3-VL models achieve higher accuracy on OmniGUIRewardBench than proprietary models such as GPT-5 and Gemini-3, removing the need for expensive closed-source APIs.

“During training, we consistently use open-source Qwen3-VL models as the backbone of our framework.They achieve higher accuracy on OmniGU-IRewardBench than proprietary models such as GPT-5 and Gemini-3, which removes the need for expensive closed-source APIs”

Probably not a substantial claim: may not pass S3 (load-bearing).

On OmniGUIRewardBench, per-trajectory average completion tokens is 6416.8.

“Completion tokens | 6416.8”

Probably not a substantial claim: may not pass S3 (load-bearing). The quote only partly states this claim.

On OmniGUIRewardBench, per-trajectory average latency is 117.6 s.

“Latency | 117.6 s”

May combine two claims.

In experiments with 64 environment workers, collecting a batch of trajectories takes about 3420 s on average, whereas OS-Themis takes about 117.6 s on average-approximately 3% of the rollout time.

“In our experiments with 64 environment workers, collecting a batch of trajectories takes about 3420 s on average, whereas OS-Themis takes about 117.6 s on average-approximately 3% of the rollout time.”

May combine two claims.

OS-Themis achieves mean overall accuracy 81.6, precision 90.9, recall 70.4 across platforms in OmniGUIRewardBench.

“Mean 80.3 90.5 70.5 84.0 92.1 76.0 79.4 88.5 61.7 89.9 88.6 60.2 85.3 90.9 75.3 81.6 90.9 70.4”

May combine two claims.

OS-Themis w/o Reviewer achieves Overall Acc 86.9, Prec 85.7, Rec 88.4.

“OS-Themis w/o Reviewer | 86.9 85.7 88.4”

Probably not a substantial claim: may not pass S1 (asserted), S3 (load-bearing), S4 (based). May combine two claims. The quote only partly states this claim.

ZeroGUI achieves mean overall accuracy 73.9, precision 85.8, recall 57.4 across platforms in OmniGUIRewardBench.

“Mean 71.0 88.4 52.4 76.1 82.2 73.0 76.1 88.2 54.4 89.6 84.3 52.3 74.3 81.5 65.4 73.9 85.8 57.4”

Probably not a substantial claim: may not pass S3 (load-bearing).

After a rollout finishes, the trajectory evaluation takes 117.6 seconds on average.

“After a rollout finishes, the trajectory is evaluated (hatched, 117.6 s on average)”

May combine two claims.

OS-Themis w/o Assignment Goal achieves Overall Acc 86.9, Prec 84.6, Rec 90.0.

“OS-Themis w/o Assignment Goal 86.9 84.6 90.0”

Probably not a substantial claim: may not pass S3 (load-bearing).

The validation reward curves exhibit noticeable oscillations.

“the reward curves exhibit noticeable oscillations.”

May combine two claims.

Current models perform poorly on macOS tasks, with even the best-performing model achieving a success rate below 10%.

“current models perform poorly on macOS tasks: even the best-performing model achieves a success rate below 10%.”

The quote only partly states this claim.

The experimental results reveal clear and systematic trade-offs between precision and recall that stem directly from the different aggregation rules of Majority, All, and Any Voting.

“Overall, the experimental results reveal clear and systematic trade-offs between precision and recall that stem directly from the different aggregation rules.”

May combine two claims.

OS-Themis improves over the Baseline by 8.6% and 5.8% for Qwen3-VL-4B and Qwen3-VL-8B policy backbones, respectively, substantially outperforming DigiRL and ZeroGUI.

“The results show that OS-Themis improves over the Baseline by 8.6% and 5.8% for the two policy backbones, respectively, substantially outperforming DigiRL and ZeroGUI.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Across all 1,409 tasks, the total number of steps is 27,882, and the total number of milestones is 9,918, representing 35.57% of the overall steps.

“Across all 1,409 tasks, the total number of steps is 27,882, and the total number of milestones is 9,918, representing 35.57% of the overall steps (step-level percentage).”

May combine two claims.

During inference, the number of completion tokens is relatively small, and the cost is dominated by the input prompt.

“during inference, the number of completion tokens is relatively small, and the cost is dominated by the input prompt.”

May combine two claims.

With larger k, Any Voting substantially boosts recall, since it becomes less likely that all models miss a true positive, but it also admits more noisy positives, which reduces precision.

“With larger k, this strategy substantially boosts recall, since it becomes less likely that all models miss a true positive.However, the same mechanism also admits more noisy positives, which reduces precision.”

Probably not a substantial claim: may not pass S3 (load-bearing).

On OmniGUIRewardBench, per-trajectory average prompt tokens is 164624.0.

“Prompt tokens | 164624.0”

Probably not a substantial claim: may not pass S3 (load-bearing).

Validation mean reward evolves for Qwen3-VL models trained via the OS-Themis framework with different base models.

“Evolution of validation mean reward for Qwen3-VL models trained via the OS-Themis framework with different base models.”

May combine two claims.

With smaller models, the gap in precision is relatively large, whereas the difference in accuracy is limited; as reasoning capability improves with scale, the gains brought by OS-Themis expand substantially, yielding clearer performance separation.

“With smaller models, the gap in precision is relatively large, whereas the difference in accuracy is limited; however, as reasoning capability improves with scale, the gains brought by OS-Themis expand substantially, yielding clearer performance separation.”

May combine two claims.

OS-Themis w/ Assignment Goal achieves Overall Acc 88.0, Prec 92.8, Rec 82.3.

“OS-Themis w/ Assignment Goal | 88.0 92.8 82.3”

May combine two claims.

The average number of milestones per task is 7.04, with a median of 6.00 and a standard deviation of 4.16.

“The average number of milestones per task is 7.04, with a median of 6.00 and a standard deviation of 4.16.”

Probably not a substantial claim: may not pass S3 (load-bearing).

After verification, the Judge Agent correctly determines that the trajectory failed to complete the task.

“After verification, the Judge Agent correctly determines that the trajectory failed to complete the task.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The overall OGRBench dataset maintains a balanced distribution between positive and negative samples.

“the overall dataset maintains a balanced distribution between positive and negative samples.”

May combine two claims.

As k increases, All Voting increasingly suppresses false positives, leading to a steady improvement and eventual saturation in precision, while making it progressively harder to return positives, manifesting as a near-linear decline in recall.

“As k increases, this strategy increasingly suppresses false positives, leading to a steady improvement and eventual saturation in precision.At the same time, the stringent consensus constraint makes it progressively harder to return positives, which manifests as a near-linear decline in recall.”

May combine two claims.

Scaling the Judge and Verifier agents to Qwen3-VL-235B yields the most significant performance gains in overall effectiveness, with the 235B Verifier achieving the highest Accuracy (82.5%) and the Judge attaining the best F1 score (81.7%).

“scaling the Judge and Verifier agents yields the most significant performance gains in terms of overall effectiveness, with the 235B Verifier achieving the highest Accuracy (82.5%) and the Judge attaining the best F1 score (81.7%).”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

AgentRewardBench covers only the Web platform and contains 1106 samples.

“AgentRewardBench | ✗ | ✗ | ✗ | ✗ | ✓ | 1106”

Probably not a substantial claim: may not pass S3 (load-bearing).

During online RL, overly long contexts caused by multi-turn interactions were frequently encountered despite setting the maximum interaction length per task to 50 steps.

“During online RL, we set the maximum interaction length per task to 50 steps, but in practice we frequently encountered overly long contexts caused by multi-turn interactions.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Apart from the 100% peak, the distribution is relatively uniform across other percentage ranges, demonstrating that milestone selection exhibits no significant bias.

“Apart from the 100% peak, the distribution is relatively uniform across other percentage ranges, demonstrating that milestone selection exhibits no significant bias.”

May combine two claims.

OS-Themis w/ Reviewer (Critic) achieves Overall Acc 88.0, Prec 92.8, Rec 82.3.

“OS-Themis w/ Reviewer (Critic) | 88.0 92.8 82.3”

May combine two claims.

OS-Themis can serve as a reliable core component for scalable data curation, enabling the construction of an autonomous and self-evolving data collection pipeline.

“OS-Themis can serve as a reliable core component for scalable data curation, enabling the construction of an autonomous and self-evolving data collection pipeline.”

May combine two claims.

Leveraging the entire refinement process rather than only the final milestone outcomes substantially reduces false positives while maintaining high recall.

“By leveraging the entire refinement process rather than only the final milestone outcomes, this approach substantially reduces false positives while maintaining high recall.”

• AssumptionRests on an assumption: False positives in reinforcement learning reinforce incorrect behaviors and are detrimental.

May combine two claims.

If recall becomes too low, truly good trajectories are rarely rewarded, leading to sparse positives, higher gradient variance, slower learning, and potentially less stable training.

“if ρ becomes small, truly good trajectories are rarely rewarded, leading to sparse positives, higher gradient variance, slower learning, and potentially less stable training.”

• AssumptionRests on an assumption: High-precision feedback is essential for stable policy optimization.

Probably not a substantial claim: may not pass S4 (based).

Granular milestone verification effectively isolates salient signals from irrelevant noise, ensuring precise evidence collection.

“This granular verification effectively isolates salient signals from irrelevant noise, ensuring precise evidence collection.”

Probably not a substantial claim: may not pass S2 (checkable).

There is an urgent need for a comprehensive ORM evaluation benchmark to establish a reliable evaluation standard for cross-platform reward modeling.

“Consequently, there is an urgent need for a comprehensive ORM evaluation benchmark to establish a reliable evaluation standard for cross-platform reward modeling.”

Probably not a substantial claim: may not pass S4 (based).

The framework possesses the inherent capacity to provide rich supervision via structured evaluation and fine-grained process extraction.

“Although the framework possesses the inherent capacity to provide rich supervision via structured evalu-ation and fine-grained process extraction”

• AssumptionRests on an assumption: Contextual information loss and evidence dilution are significant challenges in generalist GUI reward modeling.

May combine two claims.

A high false-positive rate from the Verifier degrades overall precision and hampers RL training.

“This high false-positive rate degrades overall precision and consequently hampers RL training.”

• AssumptionRests on an assumption: High-precision feedback is essential for stable policy optimization.

May combine two claims.

Training-based critics demand expensive data construction and generalize poorly to out-of-distribution environments.

“Training-based critics learn verifiers from human feedback (Qi et al., 2024;Xu et al., 2025;Wu et al., 2025b), yet they demand expensive data construction and generalize poorly to out-of-distribution (OOD) environments.”

• cites Qi et al., 2024Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Xu et al., 2025Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Wu et al., 2025bNot matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

The Reviewer Agent corrects overly optimistic assessments and prevents wrong-sign signals.

“thereby correcting overly optimistic assessments and preventing wrong-sign signals.”

• AssumptionRests on an assumption: False positives in reinforcement learning reinforce incorrect behaviors and are detrimental.

May combine two claims.

Rule-based rewards offer high precision but suffer from limited scalability and susceptibility to reward hacking.

“Rule-based rewards rely on manual heuristics (Ye et al., 2025;Lai et al., 2025b), offering high precision but suffering from limited scalability and susceptibility to reward hacking.”

• cites Ye et al., 2025Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Lai et al., 2025bNot matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

The Selector plays a crucial role in mitigating evidence dilution, reducing verification cost, and improving the quality of evidence aggregation for reliable decision making.

“These results highlight the crucial role of the Selector in mitigating evidence dilution, reducing verification cost, and improving the quality of evidence aggregation for reliable decision making.”

May combine two claims.

Increased model capacity enables the Reviewer to exercise greater rigor in its assessments, strictly minimizing false positives and ensuring high confidence in validated trajectories.

“This indicates that the increased model capacity enables the Reviewer to exercise greater rigor in its assessments, strictly minimizing false positives and ensuring high confidence in the validated trajectories.”

May combine two claims.

Majority Voting is generally reliable as a default aggregation choice, but its recall can fluctuate depending on how the threshold interacts with k.

“This behavior suggests that Majority Voting is generally reliable as a default aggregation choice, but its recall can fluctuate depending on how the threshold interacts with k.”

May combine two claims.

Increasing the positive ratio in macOSArena would require either adding many highly similar positive cases or discarding a substantial number of negative cases, both of which would reduce the diversity and representativeness of the dataset.

“Increasing the positive ratio would require either adding many highly similar positive cases or discarding a substantial number of negative cases, both of which would reduce the diversity and representativeness of the dataset.”

Probably not a substantial claim: may not pass S4 (based).

Maintaining sufficiently high recall ρ ensures true successes are rewarded often enough to provide stable learning signal.

“Maintaining sufficiently high recall ρ ensures true successes are rewarded often enough to provide stable learning signal.”

• AssumptionThe authors state this as an assumption: The evaluator is imperfect and outputs a binary reward with a fixed true-positive rate (recall) and false-positive rate.

May combine two claims.

Most existing ORM-style critics are not open-sourced, and the few open-source ones are limited by platform-specific information and thus fail to generalize.

“For ORMstyle critics, most existing models are not opensourced (Xu et al., 2025;Ye et al., 2025), and the few open-source ones are further limited by platform-specific information (Qi et al., 2024) and thus fail to generalize.”

• cites Xu et al., 2025Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Ye et al., 2025Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Qi et al., 2024Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

Only about half of the steps in each task are critical to the final outcome, and this subset of information is enough to determine whether a trajectory is correctly executed.

“These statistics indicate that only about half of the steps in each task are critical to the final outcome, and this subset of information is enough to determine whether a trajectory is correctly executed.”

May combine two claims.

The oscillations in validation reward curves are attributable to validation being computationally expensive and typically performed only at discrete checkpoints, and to the validation set being relatively small, leading to higher statistical variance and susceptibility to task composition and rollout stochasticity.

“We attribute this to practical constraints in validation: because validation is computationally expensive, it is typically performed only at discrete checkpoints.Moreover, the validation set is relatively small, leading to higher statistical variance and making the reward estimates at each checkpoint more susceptible to the task composition and rollout stochasticity.”

May combine two claims.

If a final decision were made immediately after the Milestone Verification Module finished extracting information, it would result in an erroneous judgment because the information has not been fully utilized.

“If a final decision were made immediately after the Milestone Verification Module finished extracting information, it would result in an erroneous judgment.This is because the information has not been fully utilized.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The explicit stopping rules with hard worst-case iteration caps provide a crisp algorithmic stopping rule and a strict worst-case iteration bound, making the overall runtime easier to reason about.

“These caps provide a crisp algorithmic stopping rule and a strict worst-case iteration bound, making the overall runtime easier to reason about.”

Classified as probably not a substantial claim. May combine two claims.

Precision reflects the purity of rewarded samples; if many failures are mistakenly rewarded, the update direction is contaminated and the policy may be pushed toward incorrect behaviors, especially harmful when scaling up training.

“Precision therefore reflects the purity of rewarded samples: if many failures are mistakenly rewarded, the update direction is contaminated and the policy may be pushed toward incorrect behaviors, which becomes especially harmful when scaling up training.”

• AssumptionRests on an assumption: In policy-gradient-based RL, the reward signal determines which trajectories are reinforced.

May combine two claims.

Evaluation starts immediately after each rollout finishes to compute accuracy, is decoupled from the online environment, and does not block subsequent rollouts, thus having negligible impact on RL efficiency.

“evaluation starts immediately after each rollout finishes to compute accuracy.Importantly, this evaluation is decoupled from the online environment and does not block subsequent rollouts, thus having negligible impact on RL efficiency.”

• AssumptionRests on an assumption: High-precision feedback is essential for stable policy optimization.

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

Because the prompt contains substantial repeated prefixes, vLLM's prefix caching can significantly improve throughput and reduce redundant computation overhead.

“Since the prompt contains substantial repeated prefixes (e.g., system prompts), vLLM's prefix caching can significantly improve throughput and reduce redundant computation overhead.”

Classified as probably not a substantial claim. May combine two claims.

Recall controls how often truly good behaviors are recognized and rewarded; if recall is too low, positive signal becomes sparse, slowing iteration and limiting the attainable performance ceiling.

“Recall, by contrast, controls how often truly good behaviors are recognized and rewarded; if recall is too low, positive signal becomes sparse, slowing iteration and limiting the attainable performance ceiling.”

• AssumptionRests on an assumption: In policy-gradient-based RL, the reward signal determines which trajectories are reinforced.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Introducing an additional format reward as a penalty term (0 when output format is correct, -1 otherwise) explicitly constrains output parseability and consistency, reducing training noise due to format deviations.

“we introduced an additional format reward as a penalty term on top of the outcome reward: it is 0 when the output format is correct, and -1 otherwise.This explicitly constrains output parseability and consistency, reducing training noise due to format deviations.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Through continuous practical testing using the Qwen3-VL series and iterative refinements based on errors encountered during actual experiments, a stable and reliable prompt was ultimately established.

“Through continuous practical testing using the Qwen3-VL series and iterative refinements based on errors encountered during actual experiments, we ultimately established a stable and reliable prompt.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Precision depends on the policy base rate according to Prec(θ) = p(θ)ρ / (p(θ)ρ + (1-p(θ))α).

“Precision depends on the policy base rate: Prec(θ) ≜ Pr(τ = g | r = 1).(12) By Bayes' rule, Prec(θ) = p(θ)ρ p(θ)ρ + 1 -p(θ) α .(13)”

• AssumptionThe authors state this as an assumption: The evaluator is imperfect and outputs a binary reward with a fixed true-positive rate (recall) and false-positive rate.

May combine two claims.

Under more complex and evidence-heavy settings, the quality of evidence organization and aggregation becomes a key determinant of final performance, and stronger models can better leverage the structured advantages of OS-Themis.

“This trend suggests that under more complex and evidence-heavy settings, the quality of evidence organization and aggregation becomes a key determinant of final performance, and stronger models can better leverage the structured advantages of OS-Themis.”