CoChat
Sign Up
PreprintCited by 192Artificial Intelligence ApplicationsTopic ModelingEthics and Social Impacts of AI

The DOI matches its DataCite record; 15 of the 16 references it cites were found in a registry or an index, and 1 are not checked yet; 2 of the 7 sources its claims cite were checked for support.

GDPval: Evaluating AI Model Performance on Real-World Economically Valuable Tasks

Open access
Patwardhan, Tejal, Dias, Rachel, Proehl, Elizabeth, Kim, Grace, Wang, Michele, Watkins, Olivia, Fishman, Simón Posada, Aljubeh, Marwan, Thacker, Phoebe, Fauconnet, Laurance, Kim, Natalie S., Chao, Patrick, Miserendino, Samuel, Chabot, Gildas, Li, David, Sharman, Michael, Barr, Alexandra, Glaese, Amelia, Tworek, Jerry·2025·arXiv
DOI 10.48550/arxiv.2510.04374arXiv 2510.04374
Open this paper in CoChat — free accountOpen PDF

Abstract

We introduce GDPval, a benchmark evaluating AI model capabilities on real-world economically valuable tasks. GDPval covers the majority of U.S. Bureau of Labor Statistics Work Activities for 44 occupations across the top 9 sectors contributing to U.S. GDP (Gross Domestic Product). Tasks are constructed from the representative work of industry professionals with an average of 14 years of experience. We find that frontier model performance on GDPval is improving roughly linearly over time, and that the current best frontier models are approaching industry experts in deliverable quality. We analyze the potential for frontier models, when paired with human oversight, to perform GDPval tasks cheaper and faster than unaided experts. We also demonstrate that increased reasoning effort, increased task context, and increased scaffolding improves model performance on GDPval. Finally, we open-source a gold subset of 220 tasks and provide a public automated grading service at evals.openai.com to facilitate future research in understanding real-world model capabilities.

Verified 2026-10-08 · 1 reference not checked yet

Overview

AI snapshot

Written from the paper’s full text.

TL;DR

GDPval is a new benchmark of 1,320 real-world economically valuable tasks across 44 occupations in the top 9 U.S. GDP sectors, graded primarily by blinded expert pairwise comparisons. Frontier models are improving roughly linearly and approaching expert deliverable quality on the 220-task gold subset, with Claude Opus 4.1 reaching 47.6% wins/ties against human deliverables.

Key findings

  • On the GDPval gold subset, Claude Opus 4.1 was the best-performing model: 47.6% of its deliverables were graded as better than or as good as the human deliverable. GPT-5 excelled on accuracy, while Claude Opus 4.1 excelled on aesthetics.
  • Frontier model performance on GDPval improved roughly linearly over time. Under a “try n times, then fix it” setup, GPT-5 had a 39.0% win rate with 1.39× speed improvement and 1.63× cost improvement; naive ratios were 90× speed and 474× cost.
  • The experimental automated grader achieved 66% agreement with human expert graders, only 5% below human expert inter-rating agreement of 71%. In appendix sweeps, average human-automated grader agreement was 65.7% and human inter-rater agreement was 70.8%.
  • Increased reasoning effort, increased task context, and increased scaffolding improved model performance. Prompting eliminated black-square artifacts from GPT-5 PDFs and reduced egregious PowerPoint formatting errors from 86% to 64%, while improving human preference win rates by 5 percentage points.

Methods

  • GDPval covers 9 sectors and 44 occupations that collectively earn $3T annually. The full set has 1,320 tasks with at least 30 tasks per occupation; the open-sourced gold subset has 220 tasks with 5 tasks per occupation. Tasks were created by industry experts averaging 14 years of experience and went through an average of five human reviews.
  • Primary evaluation used blinded expert pairwise comparisons of model deliverables against human expert deliverables. Models evaluated included GPT-4o, o4-mini, o3, GPT-5, Claude Opus 4.1, Gemini 2.5 Pro, and Grok 4. For plots, each model was sampled 3 times per prompt and graded by 3 different human graders, yielding 9 comparisons per prompt per model across 220 tasks.
  • Analyses included win rates overall and by sector, occupation, deliverable type, and task duration; speed and cost savings scenarios; failure-mode clustering from expert justifications; and experiments varying reasoning effort, prompting, scaffolding, and task context.

Limitations

  • Paper-stated limitations: the full set has only 44 occupations and 30 tasks per occupation; it focuses on self-contained digital knowledge work and excludes manual labor, physical tasks, extensive tacit knowledge, personally identifiable information, proprietary software, and communication between individuals. Tasks are precisely specified and one-shot rather than interactive. The automated grader has limitations compared with human experts, and constructing/running the evaluation is expensive.
  • Text-evident limitations: despite blinding, model samples may have been identifiable due to stylistic differences, such as OpenAI outputs often using em dashes, Claude outputs frequently adopting first-person phrasing, and Grok occasionally referring to itself as Grok. Cost estimates were not obtained for Claude, Gemini, and Grok. The automated grader cannot grade some tasks due to lack of internet access, Python-only execution, font package differences, and limited speech-to-text functionality.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

DataCite · 10.48550/arxiv.2510.04374

Cites

15 of 16 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Verification

Claims the paper makes

2 of 7 cited sources checked · 1 supported · 0 flagged · accuracy not yet measured

  • Premisecitation
    § 1 INTRODUCTION

    Historical evidence from technological shifts shows that the transition from invention to economy-wide permeation often takes years or even decades, requiring regulatory, cultural, and procedural changes.

    “historical evidence from technological shifts-such as electricity, airplanes, and computers-shows that the transition from invention to economy-wide permeation often takes years or even decades, requiring regulatory, cultural, and procedural changes”
    Quote in the paperSubstantial claim1 of 3 sources checked
    • cites David, 1990

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Brynjolfsson & Hitt, 2000 source

      The paper’s Abstract (page unavailable) describes earlier technologies whose benefits depended on complementary innovations, including organizational change. “Transforming the Firm” (page unavailable) also describes changes in processes and culture accompanying computer adoption. The passages do not establish a years-or-decades interval from invention to economy-wide adoption, or identify regulatory changes or airplanes as examples. (LLM, conf=0.84)

      Experimental — accuracy not yet measured
    • cites Solow, 1987

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Premisecitation
    § A.6.2 AUTOMATED GRADER CORRELATION RESULTS

    Models often favor their own responses.

    “This aligns with empirical evidence that models often favor their own responses Panickssery et al. (2024).”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites Panickssery et al. (2024) source
  • Findingown data
    § Abstract

    The current best frontier models are approaching industry experts in deliverable quality.

    “and that the current best frontier models are approaching industry experts in deliverable quality.”
    • Also stated in: another passage · plain
      “the current best frontier models are approaching industry experts in deliverable quality.”
    • Also stated in: another passage · hedged
      “models are beginning to approach parity with industry experts on the GDPval gold subset.”
    • Also stated in: another passage · plain
      “Model deliverables outperformed or matched expert humans' deliverables in just over half the tasks.”
    Quote in the paperSubstantial claim
  • Findingown data
    § A.4.3 Task Specification

    89.07% of tasks were rated as well-specified.

    “89.07% of tasks were rated as well-specified, indicating the instructions closely matched real-world expectations of clarity and detail.”
    • Also stated in: another passage · plain
      “89.07% of tasks were rated as well-specified”
    Quote in the paperNot a substantial claim
  • Findingown data
    § ABSTRACT

    Increased task context improves model performance on GDPval.

    “We also demonstrate that increased reasoning effort, increased task context, and increased scaffolding improves model performance on GDPval.”
    • Also stated in: another passage · boosted
      “the experiment in the 'Under-contextualized GDPval' section (section A.2.7) demonstrates how model performance degrades with less context.”
    Quote in the paperNot a substantial claim
  • The quote only partly states this claim. 3
  • It may state two separate claims as one. 33
  • Probably not a substantial claim. 46

  • The quote only partly states this claim.

    Findingown data
    § Abstract

    Frontier model performance on GDPval is improving roughly linearly over time.

    “We find that frontier model performance on GDPval is improving roughly linearly over time”
    • Also stated in: another passage · plain
      “Performance of OpenAI frontier models increased roughly linearly over time on the GDPval gold subset.”
    Quote in the paperNot a substantial claim

  • It may state two separate claims as one.

    Findingown data
    § ABSTRACT

    GDPval covers the top 9 sectors contributing to U.S. GDP and 44 occupations.

    “GDPval covers the majority of U.S. Bureau of Labor Statistics Work Activities for 44 occupations across the top 9 sectors contributing to U.S. GDP (Gross Domestic Product).”
    • Also stated in: another passage · plain
      “GDPval covers the top 9 sectors contributing to U.S. GDP (Gross Domestic Product), with at least 30 tasks per occupation in the full set (and 5 tasks per occupation in the gold subset), across 44 occupations.”
    • Also stated in: another passage · plain
      “GDPval covers tasks from 9 sectors and 44 occupations that collectively earn $3T annually.”
    Quote in the paperSubstantial claim

  • It may state two separate claims as one.

    Findingown data
    § 1 INTRODUCTION

    The full GDPval set contains 1,320 tasks, with at least 30 tasks per occupation; the gold subset contains 220 tasks, with 5 tasks per occupation.

    “with at least 30 tasks per occupation in the full set (and 5 tasks per occupation in the gold subset), across 44 occupations.”
    • Also stated in: another passage · plain
      “All 1,320 tasks in the full GDPval set went through an iterative review pipeline”
    • Also stated in: another passage · plain
      “the 220 open-sourced gold subset”
    Quote in the paperNot a substantial claim

  • It may state two separate claims as one.

    Findingown data
    § 2.5 HUMAN EXPERT GRADING AND AUTOMATED GRADING

    The automated grader achieves 66% agreement with human expert graders, which is only 5% below human expert inter-rating agreement of 71%.

    “the automated grader is faster and cheaper than expert grading, and achieves 66% agreement with human expert graders, only 5% below human expert inter-rating agreement of 71%.”
    • Also stated in: another passage · plain
      “We find that automated grader agreement is within 5% of human inter-rater agreement on the GDPval gold subset.”
    Quote in the paperSubstantial claim

  • It may state two separate claims as one.

    Findingown data
    § 3.1 HEADLINE RESULTS

    Claude Opus 4.1 was the best performing model on the GDPval gold subset, excelling in particular on aesthetics.

    “Claude Opus 4.1 was the best performing model on the GDPval gold subset, excelling in particular on aesthetics (e.g., document formatting, slide layout)”
    Quote in the paperSubstantial claim

  • The quote only partly states this claim.

    Findingnone given
    § ABSTRACT

    GDPval covers the majority of U.S. Bureau of Labor Statistics Work Activities for each occupation.

    “GDPval covers the majority of U.S. Bureau of Labor Statistics Work Activities for 44 occupations across the top 9 sectors contributing to U.S. GDP (Gross Domestic Product).”
    • Also stated in: another passage · plain
      “sourced to cover the majority of Work Activities tracked by O*NET for each occupation”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim.

    Findingnone given
    § 1 INTRODUCTION

    Tasks require manipulating a variety of formats (e.g., CAD design files, photos, video, audio, social media posts, diagrams, slide decks, spreadsheets, and customer support conversations).

    “Tasks require manipulating a variety of formats (e.g., CAD design files, photos, video, audio, social media posts, diagrams, slide decks, spreadsheets, and customer support conversations).”
    Quote in the paperSubstantial claim

  • It may state two separate claims as one.

    Findingown data
    § 3.3 MODEL STRENGTHS AND WEAKNESSES

    GPT-5 and Grok showed the fewest accuracy errors, though all models sometimes hallucinated data or miscalculated.

    “GPT-5 and Grok showed the fewest accuracy errors, though all models sometimes hallucinated data or miscalculated.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.

    Findingnone given
    § 2.5 HUMAN EXPERT GRADING AND AUTOMATED GRADING

    The automated grader is faster and cheaper than expert grading.

    “the automated grader is faster and cheaper than expert grading”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim. The paper may give no reason why it holds.

    Findingnone given
    § 1 INTRODUCTION

    Tasks require an average of 7 hours of work for an expert professional to complete.

    “Tasks require an average of 7 hours of work for an expert professional to complete.”
    Quote in the paperNot a substantial claim

  • It may state two separate claims as one.

    Findingown data
    § 3.3 MODEL STRENGTHS AND WEAKNESSES

    Gemini and Grok frequently promised but failed to provide deliverables, ignored reference data, or used the wrong format.

    “Gemini and Grok frequently promised but failed to provide deliverables, ignored reference data, or used the wrong format.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim. The paper may give no reason why it holds.

    Findingnone given
    § 1 INTRODUCTION

    In addition to correctness, expert graders often consider subjective factors such as structure, style, format, aesthetics, and relevance.

    “In addition to correctness, expert graders often consider subjective factors such as structure, style, format, aesthetics, and relevance.”
    Quote in the paperNot a substantial claim

  • It may state two separate claims as one.

    Findingown data
    § 3.3 MODEL STRENGTHS AND WEAKNESSES

    Claude, Grok, and Gemini most often lost due to instruction-following failures, while GPT-5 high lost mainly from formatting errors and had the fewest instruction-following issues.

    “Claude, Grok, and Gemini most often lost due to instruction-following failures, while GPT-5 high lost mainly from formatting errors and had the fewest instruction-following issues.”
    Quote in the paperNot a substantial claim

  • It may state two separate claims as one.

    Findingown data
    § 3.1 HEADLINE RESULTS

    GPT-5 performs better on pure text, while Claude performs better on file types like .pdf, .xslx, and .ppt, demonstrating better visual and aesthetic abilities.

    “GPT-5 performs better on pure text, while Claude performs better on file types like .pdf, .xslx, and .ppt, demonstrating better visual and aesthetic abilities”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim. The paper’s conclusions may not depend on it.

    Findingown data
    § 2.5 HUMAN EXPERT GRADING AND AUTOMATED GRADING

    On average, grading each comparison for the gold subset took over an hour.

    “On average, grading each comparison for the gold subset took over an hour.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim. The paper’s conclusions may not depend on it.

    Findingnone given
    § 1 INTRODUCTION

    Each task requires parsing through up to 17 reference files in the gold subset, and 38 in the full set.

    “Each task also requires parsing through up to 17 reference files in the gold subset, and 38 in the full set.”
    Quote in the paperNot a substantial claim

  • It may state two separate claims as one.

    Findingown data
    § 3.4 INCREASING REASONING EFFORT AND SCAFFOLDING

    Prompting fully eliminated black-square artifacts from GPT-5 responses, which previously affected over half of generated PDFs, and reduced egregious formatting errors in PowerPoint files from 86% to 64%.

    “Prompting fully eliminated black-square artifacts from GPT-5 responses, which previously affected over half of generated PDFs, and reduced egregious formatting errors in PowerPoint files from 86% to 64%.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim. The paper’s conclusions may not depend on it.

    Findingnone given
    § 5 LIMITATIONS

    Constructing and running the evaluation is expensive, particularly with industry expert graders.

    “Constructing and running our evaluation is expensive, particularly with industry expert graders.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim. The paper’s conclusions may not depend on it.

    Findingown data
    § A.2.7 UNDER-CONTEXTUALIZED GDPVAL

    On average, the revised under-contextualized prompts were 42% the length (by token count) of the original prompts.

    “On average, these revised prompts were 42% the length (by token count) of the original prompts.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim. The paper’s conclusions may not depend on it.

    Findingown data
    § A.2.6 ADDITIONAL DETAIL ON MODEL FAILURES ANALYSIS

    Roughly 3% of GPT-5 model failures were marked as catastrophic.

    “with roughly 3% of failures marked as catastrophic”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim. The paper’s conclusions may not depend on it.

    Findingown data
    § A.2.3 WIN RATES BY OCCUPATION

    Other occupations display near parity among multiple models.

    “others display near parity among multiple models”
    Quote in the paperNot a substantial claim

  • It may state two separate claims as one.

    Findingown data
    § A.4.2 O*NET TASKS, SKILLS, AND WORK ACTIVITIES

    The GDPval dataset covered 208 unique O*NET tasks, 25 occupational skills, and 26 work activities.

    “The dataset covered 208 unique O*NET tasks, 25 occupational skills, and 26 work activities.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim. The paper’s conclusions may not depend on it.

    Findingnone given
    § A.4.4 TASK REPRESENTATIVENESS

    Less than 10% of applicants were selected to contribute tasks to the full set.

    “Less than 10% of applicants were selected to contribute tasks to our full set.”
    Quote in the paperNot a substantial claim

  • It may state two separate claims as one.

    Findingown data
    § A.7.1 VALIDATING THE DIGITAL TASKS MEASURE

    Occupations with higher digital-task shares score systematically higher on non-routine cognitive dimensions and lower on manual dimensions.

    “The patterns are clear: occupations with higher digital-task shares score systematically higher on the non-routine cognitive dimensions and lower on the manual dimensions.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim. The paper’s conclusions may not depend on it.

    Findingown data
    § A.6.3 AUTOMATED GRADER LIMITATIONS

    In the open-source set, 12 out of 220 tasks are marked as ungradable due to limitations of the automated grader.

    “In the open-source set we mark 12 out of 220 tasks as ungradable due to limitations of the automated grader.”
    Quote in the paperNot a substantial claim

  • It may state two separate claims as one.

    Findingown data
    § A.7.1 VALIDATING THE DIGITAL TASKS MEASURE

    Knowledge-work occupations cluster at the top of the non-routine cognitive distributions and at the bottom of the routine and manual distributions.

    “The pattern is again clearknowledge-work occupations cluster at the top of the non-routine cognitive distributions and at the bottom of the routine and manual distributions.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. It may state two separate claims as one.

    Study design and conductnone given
    § 3.4 INCREASING REASONING EFFORT AND SCAFFOLDING

    The prompt applies generally to multimodal economic tasks and is not overfit to any given question.

    “The prompt applies generally to multimodal economic tasks and is not overfit to any given question”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim. It may state two separate claims as one.

    Interpretationreasoning
    § 1 INTRODUCTION

    The primary metric is win rate, which allows for continuous evaluation.

    “our primary metric is win rate, which allows for continuous evaluation.”
    Quote in the paperSubstantial claim

  • It may state two separate claims as one.

    Study design and conductown data
    § 2.4 TASK QUALITY CONTROL PIPELINE

    All 1,320 tasks in the full GDPval set went through an iterative review pipeline involving both automated model-based screening and multiple stages of human expert review, with each task receiving an average of five human reviews (minimum of three).

    “All 1,320 tasks in the full GDPval set went through an iterative review pipeline involving both automated model-based screening and multiple stages of human expert review. Each task received an average of five human reviews (with a minimum of three reviews).”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.

    Study design and conductreasoning
    § 3.1 HEADLINE RESULTS

    Comparisons were aimed to be as blind as possible, but model samples may still have been identifiable due to stylistic differences.

    “We aimed to keep comparisons as blind as possible, but model samples may still have been identifiable due to stylistic differences.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim. The paper’s conclusions may not depend on it.

    Interpretationreasoning
    § 1 INTRODUCTION

    The dataset serves as a helpful testbed to assess automated grader performance.

    “Our dataset also therefore serves as a helpful testbed to assess automated grader performance.”
    Quote in the paperNot a substantial claim

  • It may state two separate claims as one.

    Interpretationreasoning
    § 1 INTRODUCTION

    AI capability evaluations can provide clearer, more directly attributable evidence about model abilities, allowing assessment of economic relevance ahead of widespread adoption.

    “AI capability evaluations can provide clearer, more directly attributable evidence about model abilities, allowing us to assess economic relevance ahead of widespread adoption.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim. The paper’s conclusions may not depend on it.

    Study design and conductown data
    § 2.2 EXPERT RECRUITMENT We recruited expert industry professionals to create realistic tasks based on their professional work experience. Experts were required to have a minimum of 4 years of professional experience in their occupation and a strong resume with a demonstrated history of professional recognition, promotion, and management responsibilities. The average expert had 14 years of experience. We further required experts to pass a video interview, a background check, a training and a quiz to participate in the project. Experts were well compensated for their time and experience. Some of the prior employers of our industry experts include: Accenture, Aetna, Apple, AXA Advisors, Bank of America, Barclays, BBC News, Boeing, Budget Rent a Car, Capital One, Centers for Disease Control and Prevention, Citigroup, Cond´ e Nast, CVS Pharmacy, U.S. Department of Defense, Disney, Douglas Elliman, E*TRADE, Federal Trade Commission, General Electric, Goldman Sachs, Google, Guggenheim Partners, HBO, IBM, JPMorgan Chase, Johnson & Johnson, Kmart, Kirkland & Ellis LLP, LinkedIn, Lockheed Martin, Macy's, Massachusetts General Hospital, Meta, Microsoft, Morgan Stanley, National Park Service, NFL Network, Oracle, Paul, Weiss, Rifkind, Wharton & Garrison LLP, Prudential, PwC, Raytheon, Sally Beauty, Samsung, SAP, Scientific American, Sotheby's, Telegraph Media Group, Thermo Fisher Scientific, TIME, Twilio, U.S. Department of Justice, United States Air Force, United States Postal Service, Walgreens, Wells Fargo, White & Case LLP, and Whole Foods. 1 Weassigned occupations to sectors by using the 2023 BLS National Employment Matrix from U.S. Bureau of Labor Statistics (2025a) to map occupations to sectors by identifying the sector with the highest employment for each occupation. For more detail, see section A.7.

    Experts were required to pass a video interview, a background check, a training and a quiz to participate in the project.

    “We further required experts to pass a video interview, a background check, a training and a quiz to participate in the project.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim.

    Study design and conductown data
    § 2.3 TASK CREATION

    Occupational experts rated each task on its difficulty, representativeness, time to complete, and overall quality against real-world standards for their occupation.

    “To assess task quality, we asked occupational experts to rate each task on its difficulty, representativeness, time to complete, and overall quality against real-world standards for their occupation.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim. It may state two separate claims as one.

    Study design and conductnone given
    § 2.2 EXPERT RECRUITMENT We recruited expert industry professionals to create realistic tasks based on their professional work experience. Experts were required to have a minimum of 4 years of professional experience in their occupation and a strong resume with a demonstrated history of professional recognition, promotion, and management responsibilities. The average expert had 14 years of experience. We further required experts to pass a video interview, a background check, a training and a quiz to participate in the project. Experts were well compensated for their time and experience. Some of the prior employers of our industry experts include: Accenture, Aetna, Apple, AXA Advisors, Bank of America, Barclays, BBC News, Boeing, Budget Rent a Car, Capital One, Centers for Disease Control and Prevention, Citigroup, Cond´ e Nast, CVS Pharmacy, U.S. Department of Defense, Disney, Douglas Elliman, E*TRADE, Federal Trade Commission, General Electric, Goldman Sachs, Google, Guggenheim Partners, HBO, IBM, JPMorgan Chase, Johnson & Johnson, Kmart, Kirkland & Ellis LLP, LinkedIn, Lockheed Martin, Macy's, Massachusetts General Hospital, Meta, Microsoft, Morgan Stanley, National Park Service, NFL Network, Oracle, Paul, Weiss, Rifkind, Wharton & Garrison LLP, Prudential, PwC, Raytheon, Sally Beauty, Samsung, SAP, Scientific American, Sotheby's, Telegraph Media Group, Thermo Fisher Scientific, TIME, Twilio, U.S. Department of Justice, United States Air Force, United States Postal Service, Walgreens, Wells Fargo, White & Case LLP, and Whole Foods. 1 Weassigned occupations to sectors by using the 2023 BLS National Employment Matrix from U.S. Bureau of Labor Statistics (2025a) to map occupations to sectors by identifying the sector with the highest employment for each occupation. For more detail, see section A.7.

    Experts were required to have a minimum of 4 years of professional experience and a strong resume with demonstrated history of professional recognition, promotion, and management responsibilities.

    “Experts were required to have a minimum of 4 years of professional experience in their occupation and a strong resume with a demonstrated history of professional recognition, promotion, and management responsibilities.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim. The paper’s conclusions may not depend on it.

    Study design and conductown data
    § 4 OPEN-SOURCING

    As a result of limitations with the automated grader, automated grading results are not provided for all tasks in the gold subset.

    “as a result of limitations with our automated grader, we don't provide automated grading results for all tasks in the gold subset.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim. The paper’s conclusions may not depend on it.

    Study design and conductnone given
    § A.2.1 SPEED AND COST ANALYSIS, CONTINUED

    The speed and cost analysis does not capture the cost of catastrophic mistakes, which can be disproportionately expensive in some domains.

    “One further limitation of this analysis is that it does not capture the cost of catastrophic mistakes, which can be disproportionately expensive in some domains.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.

    Interpretationreasoning
    § A.2.1 SPEED AND COST ANALYSIS, CONTINUED

    The try-n-times ratio over-penalizes the model because the win rate after each completion likely goes up and the review time also goes down as the professional gets more comfortable with the task.

    “We are over-penalizing the model here, because the win rate after each completion likely goes up (because the professional will adjust the prompt to the model to fix the errors) and the review time also goes down as the professional gets more comfortable with the task.”
    Quote in the paperNot a substantial claim

  • It may state two separate claims as one.

    Study design and conductown data
    § 5 LIMITATIONS

    The GDPval full set currently consists of only 44 occupations and 30 total tasks per occupation, and is therefore a limited, initial cut of knowledge work tasks, not a comprehensive evaluation of all possible occupational tasks.

    “The GDPval full set currently consists of only 44 occupations and 30 total tasks per occupation. It is therefore a limited, initial cut of knowledge work tasks, not a comprehensive evaluation of all possible occupational tasks.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim. The paper’s conclusions may not depend on it.

    Study design and conductreasoning
    § A.2.1 SPEED AND COST ANALYSIS, CONTINUED

    The speed and cost analysis does not include consideration of the time taken to review a human professional deliverable, although this would commonly occur for tasks in GDPval.

    “We do not include consideration of the time taken to review a human professional deliverable, although this would commonly occur for tasks in GDPval”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim. The paper’s conclusions may not depend on it. The quote only partly states this claim.

    Study design and conductown data
    § A.2.1 SPEED AND COST ANALYSIS, CONTINUED

    Multiple occupational reviewers independently validated the self-reported real-world task completion times, correcting errors.

    “Multiple occupational reviewers independently validated these times, correcting errors.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim. The paper’s conclusions may not depend on it.

    Interpretationnone given
    § A.2.6 ADDITIONAL DETAIL ON MODEL FAILURES ANALYSIS

    The 23% of ratings for 'model better' roughly corresponds to the level of inter-rater agreement observed in fig. 4b.

    “The 23% of ratings for 'model better' roughly corresponds to the level of inter-rater agreement we observed in fig. 4b.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. It may state two separate claims as one.

    Study design and conductnone given
    § 5 LIMITATIONS

    For GDPval, the full context of the task is provided in the prompt, but in real life it often takes effort to figure out the full context of a task and understand what to work on.

    “For GDPval, we provide the full context of the task in the prompt, but in real life it often takes effort to figure out the full context of a task and understand what to work on.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.

    Study design and conductown data
    § A.2.7 UNDER-CONTEXTUALIZED GDPVAL

    The under-contextualized experiment was run on an earlier version of the GDPval gold subset, and therefore the observed win rates do not match those in the main text of the paper.

    “this experiment was run on an earlier version of the GDPval gold subset, and therefore the observed win rates do not match those in the main text of the paper.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim. The paper’s conclusions may not depend on it.

    Study design and conductnone given
    § A.2.1 SPEED AND COST ANALYSIS, CONTINUED

    The speed and cost analysis does not include the possibility that the human deliverable is also undesirable.

    “We also do not include the possibility that the human deliverable is also undesirable.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim. The paper’s conclusions may not depend on it.

    Interpretationreasoning
    § 5 LIMITATIONS

    The automated grader proxy is not a full substitute for industry expert graders.

    “we make an automated grader proxy available, but do not consider it a full substitute for industry expert graders.”
    Quote in the paperNot a substantial claim

  • It may state two separate claims as one.

    Interpretationreasoning
    § A.2.1 SPEED AND COST ANALYSIS, CONTINUED

    As n approaches infinity, with win rate w > 0, the time savings are HT / ((MT + RT)/w) times faster and cost savings are HC / ((MC + RC)/w) times cheaper than human experts.

    “As n → ∞ , with w > 0 , the time savings are H T / (( M T + R T ) /w ) times faster and cost savings are H C / (( M C + R C ) /w ) times cheaper than human experts.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.

    Study design and conductnone given
    § 5 LIMITATIONS

    Tasks in the initial version of GDPval are oriented around knowledge work that can be performed on a computer, particularly around digital deliverables; manual labor and physical tasks are not included; tasks involving extensive tacit knowledge, access to personally identifiable information, use of proprietary software tools, or communication between individuals are out of scope.

    “Tasks in the initial version of GDPval are oriented around knowledge work that can be performed on a computer, particularly around digital deliverables. Manual labor and physical tasks are not included in the current version. Moreover, tasks that involve extensive tacit knowledge, access to personally identifiable information, use of proprietary software tools, or communication between individuals”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim. The paper’s conclusions may not depend on it.

    Study design and conductreasoning
    § A.2.1 SPEED AND COST ANALYSIS, CONTINUED

    Because human expert task completion times were self-reported, it is possible that experts under-estimated or over-estimated the time taken.

    “Because times were self-reported, it is possible that experts under-estimated or over-estimated time taken”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim. The paper may give no reason why it holds.

    Study design and conductnone given
    § 5 LIMITATIONS

    The current automated grader has a number of limitations compared to human expert graders.

    “Our current automated grader has a number of limitations compared to human expert graders.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim. The paper’s conclusions may not depend on it.

    Study design and conductreasoning
    § A.5.2 HUMAN EXPERT REVIEWERS

    Reviewers were primarily sourced from the original expert pool based on demonstrated excellence in task creation.

    “Reviewers were primarily sourced from the original expert pool based on demonstrated excellence in task creation.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.

    Study design and conductnone given
    § A.5.2 HUMAN EXPERT REVIEWERS

    Initially, researchers manually reviewed all tasks to identify experts who produced consistently high-quality tasks; these individuals were trained and promoted to reviewers.

    “Initially, our researchers manually reviewed all tasks to identify experts who produced consistently highquality tasks; these individuals were trained and promoted to reviewers.”
    Quote in the paperNot a substantial claim

  • It may state two separate claims as one.

    Study design and conductown data
    § A.6.3 AUTOMATED GRADER LIMITATIONS

    The automated grader operates in a container that only allows running Python, which prevents grading tasks requiring other languages or external dependencies.

    “The automated grader operates in a container that only allows for running Python. Because of this, we excluded 3 Software Developers tasks that require running other languages and downloading external dependencies to properly test.”
    Quote in the paperNot a substantial claim

  • It may state two separate claims as one.

    Study design and conductown data
    § A.6.3 AUTOMATED GRADER LIMITATIONS

    The automated grader does not have internet access, so tasks that strictly require internet are not possible to grade.

    “Tasks which strictly require internet (e.g., tasks that ask agents to find music online and download it) are not possible to grade because the grader does not have internet access.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim.

    Study design and conductcitation
    § A.4.4 TASK REPRESENTATIVENESS

    Experts for each occupation were required to have previous experience in that specific occupation and sector based on the O*NET occupation definitions.

    “Experts for each occupation were required to have previous experience in that specific occupation and sector based on the O*NET occupation definitions (U.S. Bureau of Labor Statistics, 2025a).”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites U.S. Bureau of Labor Statistics, 2025a

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim. The paper’s conclusions may not depend on it.

    Study design and conductnone given
    § A.5.2 HUMAN EXPERT REVIEWERS

    The research team regularly performed quality-control checks on tasks signed off by reviewers.

    “the research team regularly performed quality-control checks on tasks signed off by reviewers”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.

    Study design and conductnone given
    § A.6.3 AUTOMATED GRADER LIMITATIONS

    The automated grader has most metrically-identical fonts, but some font packages used in human deliverables cause certain deliverables to be rendered differently than they would appear on a computer with those fonts installed.

    “Although the automated grader has most metrically-identical fonts (e.g., Liberation Sans instead of Arial), some font packages used in human deliverables still causes certain deliverables to be rendered differently than they would appear on a computer that has these fonts installed.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim. The paper’s conclusions may not depend on it.

    Study design and conductreasoning
    § A.5.1 MODEL-IN-THE-LOOP TASK REVIEW

    Experts were instructed to take model feedback as a suggestion rather than a direction.

    “experts were instructed to take model feedback as a suggestion rather than a direction”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim. The paper’s conclusions may not depend on it.

    Study design and conductnone given
    § A.5.2 HUMAN EXPERT REVIEWERS

    The most skilled reviewers were further trained to become lead reviewers, responsible for identifying, mentoring, and promoting additional qualified reviewers from within the expert pool.

    “The most skilled reviewers were further trained to become lead reviewers, responsible for identifying, mentoring, and promoting additional qualified reviewers from within the expert pool.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim. The paper’s conclusions may not depend on it.

    Premisecitation
    § A.7.1 VALIDATING THE DIGITAL TASKS MEASURE

    Each Acemoglu & Autor (2011) score is computed as a composite measure of select O*NET 'Importance' scales.

    “Each score is computed as a composite measure of select O*NET 'Importance' scales.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Acemoglu & Autor (2011) source

  • Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.

    Study design and conductown data
    § A.5.3 ITERATIVE REVIEW PROCESS

    The iterative review process included an occupation-specific expert review, where an occupation-specific reviewer assessed the representativeness of the task for the occupation and confirmed that the task was possible for another member of the occupation to complete with the provided context.

    “2. Occupation-specific expert review : An occupation-specific reviewer assessed the representativeness of the task for the occupation, and confirmed that the task was possible for another member of the occupation to complete with the provided context.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim. The paper’s conclusions may not depend on it.

    Study design and conductnone given
    § A.5.2 HUMAN EXPERT REVIEWERS

    Human reviewers conducted multiple rounds of review on each task.

    “Human reviewers conducted multiple rounds of review on each task.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim. The paper’s conclusions may not depend on it.

    Study design and conductown data
    § A.5.3 ITERATIVE REVIEW PROCESS

    The iterative review process included a final iterative reviewer feedback loop, where a third expert reviewer provided iterative feedback and worked with experts until the task met rigorous quality standards.

    “3. Final iterative reviewer feedback loop : A third expert reviewer provided iterative feedback and worked with experts until the task met our rigorous quality standards.”
    Quote in the paperNot a substantial claim

  • It may state two separate claims as one.

    Interpretationreasoning
    § A.6.2 AUTOMATED GRADER CORRELATION RESULTS

    Less capable models' outputs are easier to distinguish from human deliverables and are less likely to be preferred, which explains why agreement metrics are highest for them.

    “Both agreement metrics are highest for less capable models, since their outputs are easier to distinguish from human deliverables and are less likely to be preferred.”
    • Also stated in: another passage · plain
      “Both agreement metrics are highest for less capable models, as they can be more frequently distinguished from human deliverables and are less likely to be chosen.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.

    Study design and conductreasoning
    § A.5.1 MODEL-IN-THE-LOOP TASK REVIEW

    Experts retained final responsibility for task accuracy and completeness; the model did not autonomously alter tasks.

    “Experts retained final responsibility for task accuracy and completeness; the model did not autonomously alter tasks.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.

    Study design and conductnone given
    § A.6.3 AUTOMATED GRADER LIMITATIONS

    The automated grader has limited speech-to-text functionality inside the container and struggles with non-voice sounds.

    “The automated grader has limited speech to text functionality inside the container, and struggles with non-voice sounds.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim. The paper’s conclusions may not depend on it.

    Study design and conductown data
    § A.5.3 ITERATIVE REVIEW PROCESS

    The iterative review process included a generalist initial review, where a generalist reviewer confirmed the task adhered to project requirements.

    “1. Generalist initial review : A generalist reviewer confirmed the task adhered to project requirements.”
    Quote in the paperNot a substantial claim
67 of 67

  • The paper relies on this without support.

    Many traditional evaluations rely on text-in/text-out task formats.

    “Many traditional evaluations rely on text-in/text-out task formats.”
1 of 1

Citations

Cites

Flagged first, then by citations

Research works printed but not linked: 3

  1. Daron Acemoglu. The simple macroeconomics of ai. Economic Policy , 40(121):13-58, 2025.
  2. Paul A. David. The dynamo and the computer: An historical perspective on the modern productivity paradox. American Economic Review , 80(2):355-361, 1990. URL https://econpapers.r epec.org/RePEc:aea:aecrev:v:80:y:1990:i:2:p:355-61 .
  3. Robert M. Solow. We'd better watch out. New York Times Book Review , pp. 36, July 1987. URL https://www.standupeconomist.com/pdf/misc/solow-computer-product ivity.pdf .

Cited by

Sorted by citations

Related

Metadata

Authors
Patwardhan, T., Dias, R., Proehl, E., Kim, G., Wang, M., Watkins, O., Fishman, S. P., Aljubeh, M., Thacker, P., Fauconnet, L., Kim, N. S., Chao, P., Miserendino, S., Chabot, G., Li, D., Sharman, M., Barr, A., Glaese, A., & Tworek, J.
Affiliations
OpenAI (United States) (US)
ORCID
Watkins, Olivia
Venue
arXiv
Publisher
arXiv
Published
2025
Volume
2
Issue
4
ISSN
3067-2627 (ISSN-L)
DOI
10.48550/arxiv.2510.04374
arXiv
2510.04374
OpenAlex
W7125508931
Semantic Scholar
7ac10c0a06598a32d35be39f0f937587aaffe8e5
Cited by
189
References
8
Citations by year
2026: 177 · 2025: 12
FWCI
262.78
Type
journal article
Language
en
Open access
Licence
Locations
  • SuperIntelligence - Robotics - Safety & Alignment (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University) (cc-by)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Artificial Intelligence Applications
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
Keywords
AI benchmarking, Bureau of Labor Statistics, large language models, human oversight, reasoning effort, scaffolding, automated grading

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

A language model drew the claims from the paper on 2026-10-08, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-08 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review