SkillsBench: Benchmarking How Well Agent Skills Work Across Diverse Tasks

May combine two claims.

Skills span domains: Software Development 38%, Data Analysis 22%, DevOps/Infrastructure 15%, Writing/Documentation 12%, Other 13%.

“Software Development: 38% (git workflows, code review, testing) • Data Analysis: 22% (pandas, SQL, visualization) • DevOps/Infrastructure: 15% (Docker, Kubernetes, CI/CD) • Writing/Documentation: 12% (technical writing, API docs) • Other: 13% (scientific computing, finance, etc.)”

May combine two claims.

Codex CLI has competitive raw performance (44.7% with Skills) and frequently neglects provided Skills.

“Codex CLI: Competitive raw performance (44.7% with Skills); frequently neglects provided Skills-agents acknowledge Skills content but often implement solutions independently.”

May combine two claims.

Detailed (+18.8pp) and compact (+17.1pp) Skills provide largest benefit, while comprehensive Skills hurt performance (-2.9pp).

“Detailed (+18.8pp) and compact (+17.1pp)Skills provide the largest benefit, while comprehensive Skills actually hurt performance (-2.9pp).”

May combine two claims.

16 of 84 tasks show negative Skills deltas, including taxonomy-tree-merge (-39.3pp), energy-ac-optimal-power-flow (-14.3pp), trend-anomaly-causal-inference (-12.9pp), exoplanet-detection-period (-11.4pp).

“Despite positive domain-level aggregates, 16 of 84 tasks show negative Skills deltas: taxonomy-tree-merge (-39.3pp),energy-ac-optimal-power-flow (-14.3pp),trend-anomaly-causal-inference (-12.9pp),exoplanet-detection-period (-11.4pp).”

May combine two claims.

Claude Code has highest skills utilization rate; improvements range from +13.9pp (Opus 4.6) to +23.3pp (Opus 4.5), with all Claude models consistently benefiting.

“Claude Code: Highest skills utilization rate; improvements range from +13.9pp (Opus 4.6) to +23.3pp (Opus 4.5), with all Claude models consistently benefiting.”

May combine two claims.

Only Opus 4.6 shows a modest improvement (+1.4pp) with self-generated Skills; Codex + GPT-5.2 degrades substantially (-5.6pp), and remaining models are flat or negative.

“Only Opus 4.6 shows a modest improvement (+1.4pp);Codex + GPT-5.2 degrades substantially (-5.6pp), and remaining models are flat or negative.”

May combine two claims.

Gemini CLI has highest raw performance (Gemini 3 Flash achieves 48.7% with Skills); improvements range from +13.6pp to +17.4pp.

“Gemini CLI: Highest raw performance (Gemini 3 Flash achieves 48.7% with Skills); improvements range from +13.6pp to +17.4pp.”

May combine two claims. The quote only partly states this claim.

The ecosystem analysis collected 12,847 Skills from public GitHub repositories, 28,412 from community marketplaces, and 5,891 from corporate partners.

“Public GitHub repositories tagged with”

May combine two claims. The quote only partly states this claim.

Trajectory analysis reveals two failure modes for self-generated Skills: models generate imprecise or incomplete procedures, and for high domain-knowledge tasks models fail to recognize the need for specialized Skills.

“Trajectory analysis reveals two failure modes: (1) models identify that domain-specific knowledge is needed but generate imprecise or incomplete procedures (e.g., listing”

May combine two claims.

Skills improve performance by +16.2pp on average across 7 model-harness configurations, with range +13.6pp to +23.3pp.

“Skills improve performance by +16.2pp on average across 7 model-harness configurations, but with high variance across configurations (range: +13.6pp to +23.3pp).”

May combine two claims.

Some harnesses reliably retrieve and use Skills, while others frequently acknowledge Skills content but proceed without invoking them.

“Some harnesses reliably retrieve and use Skills, while others frequently acknowledge Skills content but proceed without invoking them.”

May combine two claims.

Tasks with 2-3 Skills show largest improvement (+18.6pp), while 4+ Skills provide only +5.9pp benefit.

“Tasks with 2-3 Skills show the largest improvement (+18.6pp), while 4+ Skills provide only +5.9pp benefit.”

Probably not a substantial claim: may not pass S3 (load-bearing).

105 contributors submitted 322 candidate tasks.

“105 contributors from both academia and industry submitted 322 candidate tasks.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The largest Skills (top 1%) exceed 50 KB and typically include extensive code resources.

“The largest Skills (top 1%) exceed 50 KB and typically include extensive code resources.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Most Skills (78%) follow the standard structure with SKILL.md plus optional resources.

“Most Skills (78%) follow the standard structure with SKILL.mdplus optional resources.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Skill sizes follow a log-normal distribution with median 2.3 KB (IQR: 0.8-6.1 KB).

“Skill sizes follow a log-normal distribution with median 2.3 KB (IQR: 0.8-6.1 KB).”

May combine two claims.

Tasks showing largest improvements include mario-coin-counting (+85.7pp, from 2.9% to 88.6%), sales-pivot-analysis (+85.7pp), flood-risk-analysis (+77.1pp), sec-financial-report (+74.3pp).

“Tasks showing largest improvements: mario-coin-counting (+85.7pp,from 2.9% to 88.6%), sales-pivot-analysis (+85.7pp),flood-risk-analysis (+77.1pp),sec-financi al-report (+74.3pp).”

Probably not a substantial claim: may not pass S3 (load-bearing).

Most Skills contain very few files (median of one, concentrated below five).

“Figure 9 shows that most Skills contain very few files (median of one, concentrated below five).”

May combine two claims.

Curated Skills improve performance by +16.2pp on average; self-generated Skills provide negligible or negative benefit.

“Curated Skills (beige) improve performance by +16.2pp on average; self-generated Skills (amber) provide negligible or negative benefit.”

May combine two claims.

Self-generated Skills yield -1.3pp on average; only Opus 4.6 shows marginal improvement (+1.4pp).

“Self-generated Skills yield -1.3pp on average; only Opus 4.6 shows marginal improvement (+1.4pp).”

Probably not a substantial claim: may not pass S3 (load-bearing).

The remaining 4 RuntimeErrors are isolated Docker daemon failures (OCI cgroup timeouts, container name conflicts, image race conditions).

“The remaining 4 errors are isolated Docker daemon failures (OCI cgroup timeouts, container name conflicts, image race conditions).”

May combine two claims.

All domains show positive aggregate delta, though individual tasks within each domain may show negative effects.

“All domains show positive aggregate delta, though individual tasks within each domain may show negative effects.”

May combine two claims.

While the absolute timeout count decreases from 367 to 328, its share of failures increases from 16.1% to 18.6%.

“While the absolute timeout count decreases from 367 to 328, its share of failures increases from 16.1% to 18.6%.”

May combine two claims.

Skills shift the Pareto frontier upward, with Gemini 3 Flash and Claude Opus dominating the with-Skills frontier.

“Skills shift the Pareto frontier upward, with Gemini 3 Flash and Claude Opus dominating the with-Skills frontier.”

May combine two claims.

The grid reveals a common set of easy tasks (top, uniformly blue) solved by all models, and hard tasks (bottom, uniformly red) unsolved even with Skills.

“The grid reveals a common set of easy tasks (top, uniformly blue) solved by all models, and hard tasks (bottom, uniformly red) unsolved even with Skills.”

May combine two claims.

Claude Code + Opus 4.5 shows the largest absolute improvement (+23.3pp) and normalized gain (29.9%) among configurations.

“• Claude Code + Opus 4.5 shows the largest absolute improvement (+23.3pp) and normalized gain (29.9%), reflecting Claude Code's native Skills integration.”

May combine two claims.

RuntimeError occurred in 46 trials, dominated by scheduling-email-assistant (42 of 46), where the Dockerfile creates symlinks for Google authentication credentials that don't exist in the evaluation environment.

“RuntimeError (46 trials).Dominated by a single task: scheduling-email-assistant (42 of 46), where the Dockerfile creates symlinks for Google authentication credentials that don't exist in the evaluation environment.”

May combine two claims.

Gemini CLI's agent initialization exceeds the 360-second setup timeout on certain task environments; this does not affect other harnesses.

“Gemini CLI's agent initialization exceeds the 360-second setup timeout on certain task environments.This does not affect other harnesses.”

May combine two claims.

Opus 4.6 has the highest timeout rate (29% of its failures) but the fewest Execution failures (10%).

“Opus 4.6 has the highest timeout rate (29% of its failures) but the fewest Execution failures (10%)”

May combine two claims.

Agents consistently loaded the data correctly but failed at pivot table creation; Codex attempted manual DataFrame reshaping instead of using pivot tables.

“Agents consistently loaded the data correctly but failed at pivot table creation-Codex attempted manual DataFrame reshaping instead of using”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Temporal dynamics of Skill creation over 136 days: daily additions remained modest through late 2025, then surged to a peak of 18,904 in January 2026; the cumulative curve reflects exponential-like growth, reaching 84,192 total Skills.

“Temporal dynamics of Skill creation over 136 days.Daily additions (bars, left axis) remained modest through late 2025, then surged to a peak of 18,904 in January 2026.The cumulative curve (line, right axis) reflects exponential-like growth, reaching 84,192 total Skills.”

May combine two claims.

Compared to Figure 11, the blue region contracts substantially, confirming that Skills shift many tasks from unsolved to solved.

“Compared to Figure11, the blue region contracts substantially, confirming that Skills shift many tasks from unsolved to solved.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

VerifierTimeoutError occurred in 83 trials: 73 instances on fix-visual-stability (excluded from evaluation) and 10 on react-performance-debugging.

“VerifierTimeoutError (83 trials).73 instances on fix-visual-stability (excluded from evaluation) and 10 on react-performance-debugging.”

May combine two claims.

The majority of cells are blue, confirming broad Skill benefit; a small number of tasks show negative delta for specific models.

“The majority of cells are blue, confirming broad Skill benefit.A small number of tasks show negative delta for specific models.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Most Skills contain 1-5 files.

“Most Skills contain 1-5 files.”

Probably not a substantial claim: may not pass S3 (load-bearing).

SKILLSBENCH consists of tasks spanning 11 domains.

“SKILLSBENCH consists of tasks spanning 11 domains.”

May combine two claims.

The Self-Generated condition shows a higher failure rate (80.9%) than No Skills (78.4%) with more Timeout failures (19.9%).

“The Self-Generated condition shows a higher failure rate (80.9%) than No Skills (78.4%) with more Timeout failures (19.9%)”

May combine two claims.

All models show high cache hit rates: GPT-5.2 at 91-92%, Gemini 3 Pro at 75-76%, and Gemini 3 Flash at 63-67%.

“All models show high cache hit rates: GPT-5.2 at 91-92%, Gemini 3 Pro at 75-76%, and Gemini 3 Flash at 63-67%.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Of 322 candidate submissions from 105 contributors, 86 tasks passed all review stages and were included in the final benchmark (26.7% acceptance rate).

“Of 322 candidate submissions from 105 contributors, 86 tasks passed all review stages and were included in the final benchmark (26.7% acceptance rate).”

Probably not a substantial claim: may not pass S3 (load-bearing).

The ecosystem is documentation-heavy: markdown files dominate, followed by scripting and configuration code.

“Figure 10 confirms the ecosystem is documentation-heavy: markdown files dominate, followed by scripting and configuration code.”

May combine two claims.

Agent timeouts represent 17.8% of agent failures, affecting 922 trials across all configurations.

“Agent timeouts (AgentTimeoutError) represent 17.8% of agent failures, affecting 922 trials across all configurations.”

May combine two claims.

At official API pricing ($0.50 vs. $2.00 per 1M input tokens), Flash's 4× lower per-token cost more than offsets this higher volume, making Flash 44% cheaper per task ($0.55 vs. $0.98).

“At official API pricing ($0.50 vs. $2.00 per 1M input tokens), Flash's 4× lower per-token cost more than offsets this higher volume, making Flash 44% cheaper per task ($0.55 vs. $0.98).”

May combine two claims.

Agent Timeout is the second most common failure mode (17.8%), followed by Incomplete Solution (10.2%) and No Output Produced (7.9%).

“Agent Timeout is the second most common (17.8%), followed by Incomplete Solution (10.2%) and No Output Produced (7.9%).”

May combine two claims.

Gemini 3 Flash/Pro have the lowest timeout rates (9-10%) but higher Execution failure rates (23-25%).

“Gemini 3 Flash/Pro have the lowest timeout rates (9-10%) but higher Execution failure rates (23-25%)”

May combine two claims.

Skills reduce the overall failure rate from 78.4% to 61.1% while shifting the failure composition.

“Skills reduce the overall failure rate from 78.4% to 61.1% while shifting the failure composition.”

May combine two claims.

AgentSetupTimeoutError occurred in 35 trials, exclusively in withskills-gemini-cli, affecting 14 tasks.

“AgentSetupTimeoutError (35 trials).Exclusively in withskills-gemini-cli, affecting 14 tasks.”

May combine two claims.

Haiku 4.5 has the highest Execution failure rate (21%) and highest overall failure rate (85.2% of trials).

“Haiku 4.5 has the highest Execution failure rate (21%) and highest overall failure rate (85.2% of trials)”

Probably not a substantial claim: may not pass S3 (load-bearing).

Token distribution of SKILL.md files (n=36,338, 99.5th percentile shown): most Skills are lightweight with median ∼1.5k tokens.

“Token distribution of SKILL.mdfiles (n=36,338, 99.5th percentile shown).Most Skills are lightweight with median ∼1.5k tokens.”

May combine two claims. The quote only partly states this claim.

On flood-risk-analysis, without Skills agents attempted ad-hoc statistical approaches and achieved only 2.9% pass rate; with Skills, pass rate rose to 80.0% (+77.1pp), with all models correctly applying the USGS-standard methodology.

“Without Skills, agents attempted ad-hoc statistical approaches-e.g., simple threshold-based detection or incorrect distribution fitting-achieving only 2.9% pass rate.The curated Skill specified the Log-Pearson Type III distribution, the standard USGS methodology for flood frequency analysis, including the exact scipy function calls and parameter interpretation.With Skills, pass rate rose to 80.0%”

• AssumptionRests on an assumption: The Log-Pearson Type III distribution is the standard USGS methodology for flood frequency analysis.

May combine two claims.

With Skills providing step-by-step guidance for the openpyxl pivot table workflow, 6 of 7 models achieved ≥80% pass rate; average improvement was +85.7pp.

“With Skills providing step-by-step guidance for the openpyxl pivot table workflow, 6 of 7 models achieved ≥80% pass rate.Average improvement: +85.7pp.”

Probably not a substantial claim: may not pass S3 (load-bearing).

There were 7,308 valid trajectories across 7 configurations and 3 conditions (no Skills, with Skills, self-generated Skills).

“Total: 7,308 valid trajectories across 7 configurations and 3 conditions (no Skills, with Skills, self-generated Skills).Self-generated condition evaluated on Claude Code and Codex configurations only.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Mean token usage per trial by model and Skills condition was measured, with Codex and Gemini data from result.json and Claude Code data from session JSONL (input tokens only, output tokens unreliable).

“Mean token usage per trial by model and Skills condition (thousands of tokens).Codex and Gemini data from result.json;ClaudeCode data from session JSONL (input tokens only, output tokens unreliable).”

Probably not a substantial claim: may not pass S3 (load-bearing).

Estimated mean cost per trial at standard API pricing was computed from mean token usage, omitting Claude Code models due to incomplete output token data.

“Estimated mean cost per trial ($) at standard API pricing.Costs computed from mean token usage.Claude Code models omitted due to incomplete output token data.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Agent timeout rates by model in the with-Skills condition were measured over 420 trials per model (84 tasks × 5 trials).

“Agent timeout rates by model in the with-Skills condition (420 trials per model, 84 tasks × 5 trials).Effective rate includes retroactive timeout enforcement for Gemini CLI (see §C).”

Classified as probably not a substantial claim.

Reviewers run benchmark experiments with and without Skills across multiple agents to confirm each task provides meaningful signal about Skill efficacy.

“Reviewers run benchmark experiments with and without Skills across multiple agents to confirm each task provides meaningful signal about Skill efficacy.”

Classified as probably not a substantial claim.

Task instructions must be written by humans, not generated by language models.

“Task instructions must be written by humans, not generated by language models.”

Classified as probably not a substantial claim.

The Docker container ensures reproducibility through isolated dependencies and clean file system state.

“The container ensures reproducibility through isolated dependencies and clean file system state.”

Classified as probably not a substantial claim.

Seven frontier models are selected: GPT-5.2, Claude Opus 4.5, Claude Opus 4.6, Claude Sonnet 4.5, Claude Haiku 4.5, Gemini 3 Pro, and Gemini 3 Flash.

“We select seven frontier models: GPT-5.2 (OpenAI), Claude Opus 4.5, Claude Opus 4.6, Claude Sonnet 4.5, Claude Haiku 4.5 (Anthropic), Gemini 3 Pro, and Gemini 3 Flash (Google).”

May combine two claims.

SKILLSBENCH focuses on terminal-based, containerized tasks, so results may not directly transfer to GUI agents, multi-agent coordination, or very long-horizon workflows.

“SKILLSBENCH focuses on terminal-based, containerized tasks for reproducible evaluation, so results may not directly transfer to GUI agents, multi-agent coordination, or very long-horizon workflows.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The study tests 7 agent-model configurations over 7,308 trajectories.

“We test 7 agent-model configurations over 7,308 trajectories.”

Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

Skills must provide procedural guidance for a class of tasks, not solutions to specific instances.

“Skills must provide procedural guidance for a class of tasks, not solutions to specific instances.”

Classified as probably not a substantial claim. May combine two claims.

Skills injection increases context length, so observed gains could partly reflect more context rather than procedural structure.

“Skills injection increases context length, so observed gains could partly reflect”

May combine two claims.

Existing benchmarks measure how well a fixed agent completes tasks; SKILLSBENCH instead measures augmentation efficacy via paired evaluation.

“These benchmarks measure how well a fixed agent completes tasks.SKILLSBENCH instead measures augmentation efficacy via paired evaluation.”

Classified as probably not a substantial claim.

The reference solution validates that each task has at least one correct solution path.

“This oracle validates that each task has at least one correct solution path.”

May combine two claims.

The self-generated Skills condition suggests structure matters-models cannot reliably produce effective procedural guidance despite having the same context budget.

“Our self-generated Skills condition suggests structure matters-models cannot reliably produce effective procedural guidance despite having the same context budget”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

Reference solution must achieve 100% test pass rate; tasks with failing oracles are rejected.

“Reference solution must achieve 100% test pass rate.Tasks with failing oracles are rejected.”

May combine two claims.

SKILLSBENCH evaluates every task under both vanilla (no Skills) and Skills-augmented conditions, enabling direct measurement of Skills efficacy.

“we evaluate every task under both vanilla (no Skills) and Skills-augmented conditions, enabling direct measurement of Skills efficacy.”

Classified as probably not a substantial claim. May combine two claims.

Pass rate is the primary metric, following Terminal-Bench scoring: average binary reward across 5 trials per task, then average task-level scores using fixed denominator of 84.

“The primary metric is pass rate, following the scoring methodology of Terminal-Bench (Merrill et al., 2026): for each task, we average the binary reward across 5 trials, then average these task-level scores using a fixed denominator of 84 (the number of evaluated tasks).”

• cites Merrill et al., 2026Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The study evaluates a limited set of models and harnesses; commercial harness behavior and Skills integration can change over time.

“We also evaluate a limited set of models and harnesses; commercial harness behavior and Skills integration can change over time.”

May combine two claims.

Focused procedural guidance is more effective than exhaustive documentation; agents may struggle to extract relevant information from lengthy Skills content, and overly elaborate Skills can consume context budget without providing actionable guidance.

“This suggests that focused procedural guidance is more effective than exhaustive documentationagents may struggle to extract relevant information from lengthy Skills content, and overly elaborate Skills can consume context budget without providing actionable guidance.”

Probably not a substantial claim: may not pass S3 (load-bearing).

All models use temperature 0 for deterministic sampling.

“All models use temperature 0 for deterministic sampling.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Tasks are stratified by difficulty measured by estimated completion time by median specialists without AI tools.

“We stratify tasks by difficulty, measured by estimated completion time by individuals whom we consider median specialists for the tasks, without the assistance of AI tools.”

Classified as probably not a substantial claim. May combine two claims.

Instructions must not reference which Skills to use, so agents must discover and apply Skills autonomously.

“Instructions must not reference which Skills to use, meaning agents must discover and apply Skills autonomously.”

May combine two claims.

Skills efficacy depends not only on Skills quality but also on how the harness implements Skills.

“Skills efficacy depends not only on Skills quality but also on how the harness implements Skills.”

May combine two claims.

Large gains are observed on domains with specialized workflows or brittle formats, and smaller or negative effects when models already have strong priors and the Skill adds overhead or conflicts.

“We observe large gains on domains with specialized workflows or brittle formats, and smaller or negative effects when models already have strong priors and the Skill adds overhead or conflicts.”

Probably not a substantial claim: may not pass S4 (based).

Despite many augmentation methods, benchmarks rarely quantify their actual impact.

“Despite many augmentation methods, benchmarks rarely quantify their actual impact.”

Classified as probably not a substantial claim.

Each task's instruction is solvable by a knowledgeable human without access to the paired Skills.

“We write instructions to be solvable by a knowledgeable human without access to the paired Skills”

Classified as probably not a substantial claim.

The evaluation uses three commercial agent harnesses: Claude Code, Codex CLI, and Gemini CLI.

“We evaluate three commercial-line agents: Claude Code (Anthropic, 2025b), Codex CLI (OpenAI, 2025), and Gemini CLI (Google, 2025).”

• cites Anthropic, 2025bNot matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites OpenAI, 2025Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Google, 2025Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing).

Each submission undergoes automated validation before human review.

“Each submission undergoes automated validation before human review:”

Classified as probably not a substantial claim.

Normalized gain is defined following Hake's formulation.

“Following Hake's formulation from physics education research (Hake, 1998), we define normalized gain as:”

• cites Hake, 1998Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Classified as probably not a substantial claim.

Each task is evaluated under three conditions: no Skills, curated Skills, and self-generated Skills.

“Each task is evaluated under three conditions: no Skills, curated Skills, and self-generated Skills.”

• Also stated in: § 3.3. Skills Conditions · plain

  “We evaluate each task under three conditions:”

Classified as probably not a substantial claim. The quote only partly states this claim.

Maintainers conduct manual review evaluating five criteria: data validity, task realism, oracle quality, Skill quality, anti-cheating.

“maintainers conduct manual review evaluating five criteria: (1) data validity: input data must reflect real-world complexity; synthetic or toy data is rejected unless justified; (2) task realism: scenarios must reflect realistic professional workflows without artificial difficulty; (3) oracle quality: reference solutions should match how domain experts would solve the task; (4) Skill quality:”

Probably not a substantial claim: may not pass S3 (load-bearing).

Skills are provided as system-level context preceding the task instruction.

“We provide Skills as system-level context preceding the task instruction in SKILLSBENCH.”

May combine two claims.

The study mitigates nondeterminism and memorization with multiple runs, a leakage audit, and paired comparisons, but cannot eliminate all nondeterminism or memorization effects.

“We mitigate with multiple runs, a leakage audit ( §2.4), and paired (Skills vs. no Skills) comparisons, yet cannot eliminate all nondeterminism or memorization effects.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Original task contributors provided human time estimates, reviewed by additional domain-expert reviewers.

“Original task contributors provided human time estimates, reviewed by an additional set of reviewers from the maintainers who are experts in the same domain.”

May combine two claims.

Excessive Skills content creates cognitive overhead or conflicting guidance.

“This non-monotonic relationship suggests that excessive Skills content creates cognitive overhead or conflicting guidance.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Instruction must be human written; human label achieved on 100% of tasks.

“Instruction must be human written (we apply both human review and GPTZero review, and we achieve human label on 100% of our tasks).”

Probably not a substantial claim: may not pass S4 (based).

All success criteria must be testable through programmatic assertions.

“All success criteria must be testable through programmatic assertions.”

May combine two claims.

The benchmark ensures strict isolation and deterministic verification.

“we ensure strict isolation and deterministic verification.”

Classified as probably not a substantial claim.

The verifier executes deterministic assertions to produce a binary pass/fail outcome.

“The verifier then executes deterministic assertions to produce a binary pass/fail outcome.”

Classified as probably not a substantial claim. May combine two claims.

Concise, stepwise guidance with at least one working example is often more effective than exhaustive documentation; overly long Skills definitions can increase context burden without improving decisions.

“Our analysis suggests that concise, stepwise guidance with at least one working example is often more effective than exhaustive documentation; overly long Skills definitions can increase context burden without improving decisions.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The self-generated condition is evaluated on 5 of 7 configurations; Gemini CLI does not support this condition.

“The self-generated condition is evaluated on 5 of 7 configurations (all Claude Code models and Codex); Gemini CLI does not support this condition.”

May combine two claims.

Existing agent benchmarks evaluate raw model capabilities in isolation, answering how well a model performs task X but not how much Skills Y improve performance on task X.

“Existing agent benchmarks (Liu et al., 2023;Merrill et al., 2026;Jimenez et al., 2024;Zhou et al., 2024b;Xie et al., 2024;Koh et al., 2024;Trivedi et al., 2024;Yang et al., 2023;Chan et al., 2025;Zhuo et al., 2025) evaluate raw model capabilities in isolation, answering”

• cites Liu et al., 2023Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Merrill et al., 2026Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Jimenez et al., 2024Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

Skills may introduce conflicting guidance or unnecessary complexity for tasks models already handle well.

“These failures suggest Skills may introduce conflicting guidance or unnecessary complexity for tasks models already handle well.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

Containerization provides state isolation but not perfect determinism or immunity to training-set leakage.

“Containerization provides state isolation but not perfect determinism or immunity to training-set leakage.”

May combine two claims.

The 84 evaluated tasks with high-quality Skills represent an optimistic scenario; real-world Skill usage involves lower-quality Skills (ecosystem mean 6.2/12 vs benchmark mean 10.1/12) and imperfect Skill-task matching.

“Our 84 evaluated tasks with high-quality Skills represent an optimistic scenario.Real-world Skill usage involves lower-quality Skills (ecosystem mean: 6.2/12 vs. benchmark mean: 10.1/12) and imperfect Skill-task matching.”

May combine two claims.

Skills reduce easy failures faster than hard ones; agents that previously produced low-quality output now spend longer pursuing better solutions, sometimes exceeding time limits.

“This is because Skills reduce easy failures faster than hard ones-agents that previously produced low-quality output now spend longer pursuing better solutions, sometimes exceeding time limits.”

Classified as probably not a substantial claim. May combine two claims.

A task passes if and only if all assertions in test_outputs.py succeed (reward = 1); partial credit is not awarded in the main evaluation.

“A task passes if and only if all assertions in test_outputs.pysucceed (reward = 1).Partial credit is not awarded in the main evaluation.”

Probably not a substantial claim: may not pass S3 (load-bearing).

For multimodal tasks (audio, PPTX, video, PDF), maintainers personally inspect agent output to verify correctness beyond programmatic assertions.

“For multimodal tasks (audio, PPTX, video, PDF), maintainers personally inspect agent output to verify correctness beyond programmatic assertions.”

• AssumptionRests on an assumption: Maintainers' personal inspection of multimodal outputs reliably verifies correctness beyond programmatic assertions.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Table 9 enumerates all 86 tasks in SKILLSBENCH; two tasks are excluded from evaluation: mhc-layer-impl (requires GPU) and fix-visual-stability (verifier timeout), leaving 84 evaluated tasks.

“Table 9 enumerates all 86 tasks in SKILLSBENCH, organized by domain.Two tasks are excluded from evaluation: mhc-layer-impl (requires GPU) and fix-visual-stability (verifier timeout), leaving 84 evaluated tasks.”

May combine two claims.

Claude models have been trained with awareness of the Agent Skills specification, which may confer advantages when processing Skill-formatted instructions.

“Claude models have been trained with awareness of the Agent Skills specification (Anthropic, 2025a), which may confer advantages when processing Skill-formatted instructions.”

• cites Anthropic, 2025aNot matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing).

Results are averaged over 5 trials per task-model-condition combination.

“Results are averaged over 5 trials per task-model-condition combination.”

Classified as probably not a substantial claim. May combine two claims.

Skills uniquely combine modular packaging with procedural guidance and optional executable resources, while remaining portable across models and harnesses.

“Skills uniquely combine modular packaging with procedural guidance and optional executable resources, while remaining portable across models and harnesses.”

May combine two claims.

Skills help Gemini 3 Pro solve tasks more efficiently, with fewer exploration rounds.

“suggesting Skills help Pro solve tasks more efficiently, with fewer exploration rounds.”

Probably not a substantial claim: may not pass S3 (load-bearing).

AI-detection screening runs GPTZero on instruction.md to flag potential model-generated content.

“AI-detection screening: Runs GPTZero on instruction.mdto flag potential model-generated content.”

• AssumptionRests on an assumption: GPTZero and human review reliably detect model-generated content in instruction.md and task.toml.

Probably not a substantial claim: may not pass S3 (load-bearing).

Skills prioritize natural-language instructions over executable implementations.

“Markdown files dominate, indicating Skills prioritize natural-language instructions over executable implementations.”

May combine two claims.

The withskills-gemini-cli configuration had retry irregularities: 13 tasks had more than the expected 10 trials, and 64 launched jobs never completed.

“The withskills-gemini-cli configuration had retry irregularities: 13 tasks had more than the expected 10 trials (5 per model), and 64 launched jobs never completed (missing result.json).”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The self-generated condition is evaluated on Claude Code (all four models) and Codex (GPT-5.2) only; Gemini CLI does not support this condition due to its explicit skill-activation interface.

“The self-generated condition is evaluated on Claude Code (all four models) and Codex (GPT-5.2) only; Gemini CLI does not support this condition due to its explicit skill-activation interface.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The evaluation comprises 7,308 valid trajectories across 7 configurations, 84 tasks, and 3 conditions.

“The evaluation comprises 7,308 valid trajectories across 7 configurations, 84 tasks, and 3 conditions.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Mean pass rates are reported with 95% bootstrap confidence intervals.

“We report mean pass rates with 95% bootstrap confidence intervals.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Oracle execution (harbor run -a oracle) runs the oracle solution end-to-end and requires 100% test pass rate.

“Oracle execution (harbor run -a oracle): Runs the oracle solution end-to-end and requires 100% test pass rate.”

The quote only partly states this claim.

These tasks exceed the computational budget regardless of model capability.

“suggesting these tasks exceed the computational budget regardless of model capability”

The quote only partly states this claim.

Claude Code's native Skills integration explains why Claude Code + Opus 4.5 shows the largest improvement.

“reflecting Claude Code's native Skills integration”

May combine two claims.

Agents spend additional time generating and then attempting to use self-authored Skills documents, which may contain errors or misleading guidance.

“suggesting that agents spend additional time generating and then attempting to use self-authored Skills documents, which may contain errors or misleading guidance.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

LLM-backed quality checks include automated verification of behavior consistency between instructions and tests, anticheating measures, pinned dependency checks, typo detection, and hardcoded solution detection.

“LLM-backed quality checks: Automated verification of behavior consistency between instructions and tests, anticheating measures, pinned dependency checks, typo detection, and hardcoded solution detection.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

The 7 agent-model configurations tightly couple specific models with proprietary agent logic, representing real-world deployment conditions.

“These tightly couple specific models with proprietary agent logic, representing real-world deployment conditions.”

May combine two claims.

Gemini 3 Flash/Pro indicate faster but more error-prone execution.

“indicating faster but more error-prone execution.”

May combine two claims.

Eight Gemini with-Skills trials that scored reward > 0 were overridden to reward = 0, reducing Flash by 0.7pp and Pro by 0.7pp.

“Eight Gemini with-Skills trials that scored reward > 0 were overridden to reward = 0, reducing Flash by 0.7pp and Pro by 0.7pp.”

Classified as probably not a substantial claim. May combine two claims.

Instructions never reference which Skills to use; agents must discover and apply them autonomously.

“Instructions never reference which Skills to use-agents must discover and apply them autonomously.”

May combine two claims.

The performance improvement (+16.2pp average) substantially outweighs the marginal cost increase ($0.03-0.22 per trial).

“but the performance improvement (+16.2ppaverage) substantially outweighs the marginal cost increase ($0.03-0.22 per trial).”

May combine two claims.

Agents typically understand the task structure and produce output, but their solutions are insufficiently accurate.

“The dominant failure mode is Quality Below Threshold (49.8%), indicating that agents typically understand the task structure and produce output, but their solutions are insufficiently accurate.”

Probably not a substantial claim: may not pass S3 (load-bearing).

fix-visual-stability was excluded from all experiments due to consistent verifier failures across all configurations.

“fix-visual-stability was excluded from all experiments due to consistent verifier failures across all configurations.”

May combine two claims.

The withskills-gemini-cli configuration recorded 0 AgentTimeoutErrors (vs. 13% expected based on other configurations), indicating timeout enforcement was disabled for these runs.

“The withskills-gemini-cli configuration recorded 0 AgentTimeoutErrors (vs.13% expected based on other configurations), indicating timeout enforcement was disabled for these runs.”

May combine two claims.

For a subset of configurations (GPT-5.2 and Claude Opus 4.5, all 26 Extreme tasks, no-Skills and with-Skills conditions), 10 runs instead of 5 showed mean ∆ within 1.2pp of 5-run estimates, standard error reduced by ∼30%, and all conclusions unchanged.

“For a subset of configurations (GPT-5.2and Claude Opus 4.5, all 26 Extreme tasks, no-Skills and with-Skills conditions), we conducted 10 runs instead of 5. Results show: • Mean ∆ within 1.2pp of 5-run estimates • Standard error reduced by ∼30% • All conclusions remain unchanged”

Classified as probably not a substantial claim.

Complex multi-step pipeline tasks (6 tasks: gh-repo-analytics, pedestrian-traffic-counting, enterprise-information-search, setup-fuzzing-py, reserves-at-risk-calc, shock-analysis-supply) require chaining multiple tools, APIs, or data sources where failure at any stage prevents downstream progress.

“Complex multi-step pipelines (6 tasks): gh-repo-analytics, pedestrian-traffic-counting, enterprise-information-search, setup-fuzzing-py, reserves-at-risk-calc, shock-analysis-supply.These require chaining multiple tools, APIs, or data sources where failure at any stage prevents downstream progress.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Difficulty levels are assigned by task authors and validated during review.

“Difficulty levels are assigned by task authors and validated during review.”

Classified as probably not a substantial claim.

Scores are computed using Method D (task-mean with fixed denominator of 84 evaluated tasks), consistent with Terminal-Bench scoring methodology.

“Scores are computed using Method D (task-mean with fixed denominator of 84 evaluated tasks), consistent with Terminal-Bench (Merrill et al., 2026) scoring methodology.”

• cites Merrill et al., 2026 sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Infrastructure errors occurred across 7,414 total trials and were treated as reward = 0 in scoring.

“Infrastructure errors across 7,414 total trials (including excluded tasks).These errors are treated as reward = 0 in scoring.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Computationally intractable tasks (6 tasks: gravitational-wave-detection, quantum-numerical-simulation, seismic-phase-picking, shock-analysis-demand, speaker-diarization-subtitles, taxonomy-tree-merge) require either domain-specific signal processing pipelines, large-scale numerical optimization, or computationally expensive algorithms that exceed the agent's time budget.

“Computationally intractable (6 tasks): gravitational-wave-detection, quantum-numerical-simulation, seismic-phase-picking, shock-analysis-demand, speaker-diarization-subtitles, taxonomy-tree-merge.These tasks require either domainspecific signal processing pipelines, large-scale numerical optimization, or computationally expensive algorithms that exceed the agent's time budget.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Strict specification tasks (4 tasks: latex-formula-extraction, pddl-tpp-planning, react-performance-debugging, xlsx-recover-data) have narrow success criteria where even small deviations from the expected output format or values cause all tests to fail.

“Strict specification tasks (4 tasks): latex-formula-extraction, pddl-tpp-planning, react-performance-debugging, xlsx-recover-data.These have narrow success criteria where even small deviations from the expected output format or values cause all tests to fail.”