CoChat
Sign Up
PreprintCited by 16Robotic Process Automation ApplicationsWeb Data Mining and AnalysisInformation Retrieval and Search Behavior

WorkArena: How Capable Are Web Agents at Solving Common Knowledge Work Tasks?

Open access
Drouin, Alexandre, Gasse, Maxime, Caccia, Massimo, Laradji, Issam H., Del Verme, Manuel, Marty, Tom, Boisvert, Léo, Thakkar, Megh +4 more·2024·arXiv
DOI 10.48550/arxiv.2403.07718arXiv 2403.07718
Open this paper in CoChat — free accountOpen PDF

Abstract

We study the use of large language model-based agents for interacting with software via web browsers. Unlike prior work, we focus on measuring the agents' ability to perform tasks that span the typical daily work of knowledge workers utilizing enterprise software systems. To this end, we propose WorkArena, a remote-hosted benchmark of 33 tasks based on the widely-used ServiceNow platform. We also introduce BrowserGym, an environment for the design and evaluation of such agents, offering a rich set of actions as well as multimodal observations. Our empirical evaluation reveals that while current agents show promise on WorkArena, there remains a considerable gap towards achieving full task automation. Notably, our analysis uncovers a significant performance disparity between open and closed-source LLMs, highlighting a critical area for future exploration and development in the field.

AI snapshot

Written from the paper’s full text.

TL;DR

WorkArena is a new remote-hosted benchmark of 33 ServiceNow-based tasks with 19,912 unique instances for evaluating web agents on common enterprise knowledge-work interactions, paired with BrowserGym, a flexible environment for web-agent design and evaluation. Current LLM agents remain far from full automation, with GPT-4o reaching 42.7% success on WorkArena versus 6.1% for GPT-3.5 and 17.9% for Llama3.

Key findings

  • GPT-4o strongly outperformed other tested LLMs on WorkArena: 42.7% ± 1.5 success, compared with GPT-3.5 at 6.1% ± 1.3 and Llama3 at 17.9% ± 1.5; GPT-4o-V scored 41.8% ± 1.7.
  • WorkArena remained challenging: no agent achieved 100% success on any specific task, and list-filter tasks had 0% success for all LLMs despite being simple for humans.
  • BrowserGym features helped set state-of-the-art zero-shot results on other benchmarks: GPT-4o achieved 23.5% on WebArena, above the original paper’s 14.4% with GPT-4, and 82.9% on the WebGum subset of MiniWoB.
  • Ablations showed chain-of-thought was crucial, with a 10-point drop for Llama3 on both MiniWoB and WorkArena when removed; adding more features sometimes hurt weaker LLMs, and “use think history” could make agents stick to early erroneous decisions.

Methods

  • WorkArena comprises 33 tasks and 19,912 unique instances built on the ServiceNow platform, covering menus, lists, forms, knowledge bases, service catalogs, and dashboards; tasks include validation functions and Playwright-based oracle functions.
  • BrowserGym is an OpenAI Gym/POMDP-style environment using Chromium and Playwright, with multimodal observations (HTML DOM, accessibility tree, screenshot), augmented element attributes, chat interaction, and action spaces including bid-based, coordinate-based, and Python/Playwright actions.
  • Experiments used a chain-of-thought web agent with GPT-3.5, GPT-4o, GPT-4o-V, and Llama3-70b; 10 seeds per task for MiniWoB and WorkArena, 1 for WebArena, max 15 steps per episode, stratified bootstrap with 1,000 samples, and random search for best agent configurations.

Limitations

  • The paper states that 15 steps may be insufficient for some WorkArena tasks unless the agent runs in multi-action mode, and that prompt truncation occurs for GPT-3.5 and Llama3 due to limited context lengths.
  • The paper notes that ablation results use a different seed and therefore differ slightly from Table 2, and that human-coded oracle functions, while correct, may not correspond to optimal solutions.
  • Evident from the text: the benchmark is tied to the ServiceNow platform and a specific set of enterprise tasks, the evaluation covers only a few LLMs and one agent design, and the paper has no dedicated limitations section.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

DataCite · 10.48550/arxiv.2403.07718

Cites

29 of 30 references verified0 citation errors · 1 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

8 of 18 cited sources checked · 7 supported · 0 flagged · accuracy not yet measured

  • Findingown data

    The 23.5% WebArena score is above the 14.4% success rate obtained using GPT-4 in the original paper.

    “way above the 14.4% succes rate obtained using GPT-4 in the original paper (Zhou et al., 2023).”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites Zhou et al., 2023 source
  • Findingown data

    GPT-4o agent achieves 82.9% on the WebGum subset of MiniWoB.

    “The outcomes on the full benchmark and the WebGum subset (82.9%) surpass those of prior studies on zero-shot web agents (Assouel et al., 2023;Zeng et al., 2023)”
    Quote in the paperSubstantial claim1 of 2 sources checked
    • cites Assouel et al., 2023

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Zeng et al., 2023 source
  • Findingnone given

    BrowserGym is compatible with previous benchmarks like WebArena, MiniWoB, and WebShop.

    “BrowserGym: A new framework for the development and evaluation of web agents, compatible with previous benchmarks like WebArena (Zhou et al., 2023), MiniWoB (Liu et al., 2018;Shi et al., 2017a) and WebShop (Yao et al., 2022)”
    Quote in the paperSubstantial claim1 of 3 sources checked
    • cites Zhou et al., 2023 source
    • cites Liu et al., 2018;Shi et al., 2017a

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Yao et al., 2022

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Findingown data

    There is a drop of 10 points for Llama3 on both MiniWoB and WorkArena when chain-of-thought is removed.

    “This is clearly seen in Table 5, with a drop of 10 points for Llama3 on both MiniWoB and WorkArena.”
    Quote in the paperSubstantial claim
  • Findingcitation

    The performance disparity between LLMs is significantly more pronounced in WorkArena and WebArena than in MiniWoB.

    “The performance disparity between LLMs is significantly more pronounced in WorkArena and WebArena than in MiniWoB, aligning with findings on the emergent properties of AI systems (Wei et al., 2022a).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Wei et al., 2022a

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Classified as probably not a substantial claim. 47
  • May combine two claims. 28
  • The quote only partly states this claim. 2

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductreasoning

    The study uses a single generic example of how chain of thought and action outputs should be formatted, aligning with objective of developing zero-shot agents.

    “In the prompt, we provide a single generic example of how the chain of thought and action outputs should be formatted.This contrasts with other methods (Kim et al., 2023) where task-specific few-shot examples are provided, yet aligns with our objective of developing zero-shot agents able to solve a large range of new tasks.”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites Kim et al., 2023 source

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data

    BrowserGym currently supports MiniWoB, WebArena and WorkArena.

    “We currently support MiniWoB § B.2), WebArena (Zhou et al., 2023) and WorkArena.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites Zhou et al., 2023 source

  • May combine two claims.

    Findingown data

    GPT-4o agent's outcomes on MiniWoB full benchmark and WebGum subset surpass prior studies on zero-shot web agents.

    “surpass those of prior studies on zero-shot web agents (Assouel et al., 2023;Zeng et al., 2023)”
    Quote in the paperSubstantial claim1 of 2 sources checked
    • cites Assouel et al., 2023

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Zeng et al., 2023 source

      The Evaluation Setup reports prior agent results on MiniWoB++ and WebArena, providing a basis for comparison with the citing paper’s results. It does not report WebGum results, and its task overview lists varying numbers of shots rather than establishing a uniformly zero-shot comparison. GPT-4o’s outcomes are the citing paper’s own results, not a finding attributed to Zeng et al. (LLM, conf=0.83)

      Experimental — accuracy not yet measured

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingcitation

    BrowserGym's observation space contains chat content, open pages, error message, and a multi-modal view of the active web page (HTML DOM, accessibility tree, screenshot).

    “at each time step, the observation space contains the content of the chat (list of messages), the currently open pages (list of URLs), the error message from the last action if any (stack trace), and a multi-modal view of the active web page: its HTML DOM snapshot (structured object), its accessibility tree or AXTree (structured object) as originally proposed by Zhou et al. (2023), and a viewport”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites Zhou et al. (2023) source

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductcitation

    The study uses GPT-4o vision and Set-of-Mark as proposed in WebVoyager.

    “We also explore the effect of providing the screenshot of the page using GPT-4o vision and Set-of-Mark (Yang et al., 2023) as proposed in WebVoyager (He et al., 2024).”
    Quote in the paperNot a substantial claim1 of 2 sources checked
    • cites Yang et al., 2023 source
    • cites He et al., 2024

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • The quote only partly states this claim.

    Findingown data

    List-based tasks in WorkArena have a 0% success rate for all LLMs.

    “Notably, the list-based tasks which require interacting with a non-standard HTML widget ( § A.2 Figure 5) prove to be the most challenging ones with a clear 0% success rate for all LLMs”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: List-based tasks are fairly simple for a human to complete.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data

    BrowserGym's action space includes Python code, high-level primitives, and coord-based actions.

    “the action space is customizable and includes Python code, which can be restricted to specific set of pre-defined high-level primitives, such as bid-based actions (click(bid), type(bid,text), . . .), and coord-based actions (mouse click(x,y), keyboard type(text), . . .).”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given

    BrowserGym provides a unique identifier (bid), bounding box coordinates, and flags indicating whether an element is visible and clickable for every element on the current page.

    “For every element on the current page BrowserGym provides a unique identifier (bid), its bounding box coordinates (left,top,right,bottom), and flags indicating whether it is visible and clickable.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    Knowledge base task yields 1,000 instances.

    “In total, this task yields 1,000 instances.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    WorkArena includes an information retrieval task that consists of searching the platform's knowledge base to answer a question.

    “The benchmark includes an information retrieval task that consists of searching the platform's knowledge base to answer a question.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    Service catalog tasks yield 3,550 instances.

    “Together, these tasks yield 3,550 instances.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    WorkArena includes 4 dashboard retrieval tasks.

    “The benchmark includes 4 retrieval tasks that require extracting numerical data from charts and (optionally) performing simple reasoning over them to answer a question.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given

    Implementing a new task in BrowserGym boils down to implementing four functions: setup(), teardown(), validate() and cheat().

    “Implementing a new task in BrowserGym boils down to implementing four functions: setup(), teardown(), validate() and cheat() (an optional oracle function).”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given

    WorkArena operates independently of any proprietary code.

    “It is important to note that although our benchmark is built on top of ServiceNow's platform, it operates independently of any proprietary code.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    WorkArena includes 2 menu-based tasks.

    “We consider 2 menu-based tasks: i) navigating via the”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    Dashboard tasks yield 1,862 instances.

    “Together, these tasks yield 1,862 instances.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingnone given

    WorkArena is open-source and designed to be easily extended by the scientific community.

    “WorkArena is open-source and designed to be easily extended by the scientific community.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

    Findingnone given

    BrowserGym natively supports web tasks that require multiple open pages and is robust to nested encapsulation techniques such as iFrames and shadow DOMs.

    “BrowserGym natively supports web tasks that require multiple open pages (tabs, popups) and is also robust to web pages that employ nested encapsulation techniques such as iFrames and shadow DOMs.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingreasoning

    BrowserGym is a framework for the development and evaluation of web agents.

    “We also introduce BrowserGym, an environment for the design and evaluation of such agents, offering a rich set of actions as well as multimodal observations.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    GPT-4o achieves 43% success on WorkArena, compared to GPT-3.5's 6.1% and Llama3's 17.9%.

    “The performance gap is particularly striking on WorkArena, where GPT-4o achieves 43% success, in stark contrast to GPT-3.5's 6.1% and Llama3's 17.9%.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Llama3 agent performs poorer than closed-source GPT-4o but remarkably high compared to GPT-3.5 agent despite shorter context window (8K vs 16K).

    “While our Llama3 agent still performs poorer than a closed-source GPT-4o, its performance is remarkably high compared to a GPT-3.5 agent despite its shorter context window (8K vs 16K).”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). The quote only partly states this claim.

    Findingown data

    Menu-based tasks yield 1,600 instances.

    “Together, these tasks yield 1,600 instances.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    WorkArena includes 12 list-based tasks.

    “We consider 12 list-based tasks, which can be grouped into two categories: filtering and sorting.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given

    BrowserGym does not impose any restriction on how web agents should be implemented.

    “BrowserGym offers an extensive list of features but does not impose any restriction on how web agents should be implemented.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    Form-based tasks yield 5,000 instances.

    “Together, these tasks yield 5,000 instances.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    As more features are added, the prompt becomes longer and seems to overwhelm the LLM, mostly observed with GPT-3.5 and Llama3.

    “As we gradually add more features, the prompt becomes longer and seems to overwhelm the LLM.This is mostly observed with GPT-3.5 and Llama3, the weaker LLMs”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    WorkArena includes 5 form-based tasks.

    “We consider 5 form-based tasks, which each consist of creating a new entry in a given data table.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingnone given

    ServiceNow web pages are heavily dynamic and exhibit complex UI elements.

    “First, the web pages are heavily dynamic, and exhibit complex UI elements and ways of interacting with them.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    List-based tasks yield 6,900 instances.

    “Together, these tasks yield 6,900 instances.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Current agents show promise on WorkArena, but there remains a considerable gap towards achieving full task automation.

    “Our empirical evaluation reveals that while current agents show promise on WorkArena, there remains a considerable gap towards achieving full task automation.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    GPT-4o agent achieves 23.5% on WebArena, which is the best zero-shot performance reported so far.

    “On WebArena, while the score obtained by our GPT-4o agent is only 23.5%, it constitutes the best zero-shot performance reported so far”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    WorkArena includes 9 service catalog tasks.

    “The benchmark includes 9 tasks that require navigating a catalog of products and ordering items with given specifications.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given

    BrowserGym extracts at each time step an observation based only on the current view of the page.

    “BrowserGym extracts at each time step an observation that is based only on the current view of the page.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The coordinate-feature gain does not translate to WorkArena, where the tasks seem best solved using bid only.

    “this does not translate to WorkArena where the tasks seem best solved using bid only.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingnone given

    On both WorkArena and MiniWoB, the vast majority of tasks do not require navigation, and all the necessary information is always present on the current page.

    “On both WorkArena and MiniWoB the vast majority of tasks does not require navigation, and all the necessary information is always present on the current page.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Agents with the think-history feature activated tend to stick to decisions made in early time steps, even erroneous ones, and are less prone to self-correction, mostly observed with the GPT agents.

    “What we observe instead, is that agents with this feature activated tend to stick to decisions decided in early time steps, even erroneous ones, and are somehow less prone to self-correction.This is mostly observed with the GPT agents, Tables 3 and 4.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductown data

    The study uses a common agent code-base tuned by activating/deactivating flags, and finds best agent configuration via random search on MiniWoB and WorkArena.

    “We evaluate all LLMs using a common agent code-base that we tune by activating / deactivating flags (see § 5.1).For each LLM we find the best agent configuration via random search on MiniWoB and WorkArena.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    All experiments are run with BrowserGym v0.3.5 and WorkArena v0.3.0.

    “All experiments are run with BrowserGym v0.3.5 and WorkArena v0.3.0.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductcitation

    15 steps is considered sufficient for MiniWoB, but on WorkArena some tasks might require more than 15 steps to complete unless the agent runs in multi-action mode.

    “Note that 15 steps is considered sufficient for MiniWoB, but on WorkArena some tasks might require more than 15 steps to complete unless the agent runs in multi-action mode”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites He et al. (2024)

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductreasoning

    The number of seeds is limited to 10 per task for MiniWoB and WorkArena, and 1 per task for WebArena.

    “we limit the number of seeds to 10 per task for MiniWoB and WorkArena, and 1 per task for WebArena.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductown data

    The study uses stratified bootstrap to obtain 1,000 samples of the mean and reports the average and standard deviation as success rate and standard error.

    “We use stratified bootstrap to obtain 1,000 samples of the mean and report the average and standard deviation of these means as success rate and standard error.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductcitation

    The study uses chain-of-thought prompting.

    “We implement a simple web agent with chain-of-thought prompting (Wei et al., 2022b)”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Wei et al., 2022b

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductcitation

    The study uses a maximum of 15 steps per episode for all tasks.

    “In all tasks we give our agents a maximum of 15 steps per episode, as in He et al. (2024).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites He et al. (2024)

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductown data

    The study uses a parsing loop that can re-prompt the agent up to 4 times to make it aware of a parsing mistake.

    “Once the LLM provides an answer, we have a parsing loop that can re-prompt the agent up to 4 times to make it aware of a parsing mistake.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    Answer validation allows slight variations in formatting and wording by verifying whether the agent's answer is within a set of acceptable answers.

    “Despite the precise formatting instructions included with each question, we allow for slight variations in formatting and wording by verifying if the answer produced by the agent is within a set of acceptable answers.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning

    The poor multimodal performance could be explained by the lack of screen-related training data in current vision-language models.

    “which could be explained by the lack of screen-related training data in current vision-language models.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    The study uses flags to activate or deactivate features such as use thinking for chain-of-thought.

    “To study the effect of different design choices in our agents, we use flags to activate or deactivate certain features, such as use thinking for chain-of-thought.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductnone given

    The study uses a maximum prompt length of 40K tokens for GPT-4o, 15K for GPT-3.5 and 8K for Llama3.

    “We use a maximum prompt length of 40K tokens for GPT-4o, 15K for GPT-3.5 and 8K for Llama3.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductnone given

    WebArena is deterministic, so it was not used for tuning.

    “WebArena being deterministic, we don't use it for tuning.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductown data

    The ablation study was done with a different seed, hence its numbers differ slightly from those in Table 2.

    “The ablation study was done with a different seed, hence numbers differ slightly from those in Table2.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationown data

    The results validate WorkArena as an unsolved, challenging benchmark that requires advanced reasoning capabilities over long contexts (HTML or AXTree), which seem to emerge only in very large models.

    “Our results validate WorkArena as an unsolved, challenging benchmark that requires advanced reasoning capabilities over long contexts (HTML or AXTree), which seem to emerge only in very large models.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Interpretationnone given

    WorkArena underscores the challenge of navigating real-world enterprise websites, notably dealing with large observations and complex, non-standard interfaces.

    “WorkArena underscores the challenge of navigating real-world enterprise websites, notably dealing with large observations and complex, non-standard interfaces.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductreasoning

    When the prompt is too large, the study progressively truncates the HTML and AXTree from the end until it fits the maximum allowed number of tokens.

    “When the prompt is too large (Figure 4), we progressively truncate the HTML and AXTree from the end until it fits the maximum allowed number of tokens.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Study design and conductreasoning

    GPT-3.5 was used to answer the generated knowledge-base questions in order to avoid the pitfall of GPT-4 catering to itself by producing ambiguous questions it can answer.

    “Note that we use GPT-3.5 to answer the questions to avoid the pitfall where GPT-4 would cater to itself, producing ambiguous questions that it somehow succeeds in answering correctly.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

    Interpretationnone given

    BrowserGym allows for the flawless evaluation of web agents on multiple benchmarks within the same unified framework, and currently supports MiniWoB, WebArena and WorkArena.

    “BrowserGym allows for the flawless evaluation of web agents on multiple benchmarks within the same unified framework, and currently supports MiniWoB, WebArena and WorkArena.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductown data

    The study fixes final configuration (one per LLM) and re-runs full evaluation on all benchmarks with a different seed.

    “We then fix the final configuration (one per LLM), and re-run a full evaluation on all benchmarks with a different seed.7Final configurations are reported in Table 1.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductown data

    On WebArena and WorkArena only AXTrees are used because HTML is prohibitively large, while on MiniWoB both AXTree and HTML are used because that consistently gave the best performance.

    “On WebArena and WorkArena we only use AXTrees because HTML is prohibitively large.On MiniWoB we use both AXTree and HTML as it consistently gave the best performance.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductnone given

    Human-coded oracle functions, while correct, might not correspond to optimal solutions, so the oracle action counts are only indicative of task complexity.

    “This is indicative of task complexity, but we emphasize that human-coded oracle functions, while correct, might not correspond to optimal solutions.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationown data

    While 2D actions are clearly required in specific cases (e.g., drawing on a web whiteboard), most web interactions do not require them, which can greatly simplify the agent design.

    “This seems to indicate that, while 2D actions are clearly required in specific cases (i.e., drawing on a web whiteboard), most web interactions actually do not require it, which can greatly simplify the agent design.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The task distribution of WorkArena and MiniWoB is representative of web interactions in general, so conclusions drawn from these benchmarks extend to most web interactions.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Similarly poor multimodal performances are reported by Xie et al. (2024).

    “Similarly poor multimodal performances are reported by Xie et al. (2024)”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Xie et al. (2024)

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

60 of 60

  • The paper relies on this without support.

    Enterprise software is an unexplored area for web agents.

    “Yet, one area in which web agents can be particularly impactful and remain unexplored is enterprise software.”

  • The paper relies on this without support.

    In the workplace, repetitive tasks are common, and enterprise software often prioritizes functionality over user experience, leading to inefficiencies and long learning curves for workers.

    “In the workplace, where repetitive tasks are common, enterprise software often prioritizes functionality over user experience, leading to inefficiencies and long learning curves for workers.”

  • The paper relies on this without support.

    WorkArena tasks are representative of a wide array of common operations that employees perform daily.

    “Collectively, these tasks are representative of a wide array of common operations that employees, like IT, administrative, and white-collar staff, perform on a daily basis.”

  • The paper relies on this without support.

    List-based tasks are fairly simple for a human to complete.

    “while being fairly simple for a human to complete.”

  • The paper relies on this without support.

    The task distribution of WorkArena and MiniWoB is representative of web interactions in general, so conclusions drawn from these benchmarks extend to most web interactions.

    “This seems to indicate that, while 2D actions are clearly required in specific cases (i.e., drawing on a web whiteboard), most web interactions actually do not require it, which can greatly simplify the agent design.”

  • The paper relies on this without support.

    The GPT-4-generated knowledge base articles and questions are adequate proxies for real enterprise knowledge base content, so WorkArena's knowledge-base task validly measures knowledge work.

    “The knowledge base included in WorkArena consists of 100 articles generated using GPT-4 (OpenAI, 2023).”
6 of 6

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Frameworks such as ToolBench, WebShop, and WorkArena rely on fixed APIs or self-hosted environments that abstract away authentication, failure recovery, and long-term user or task state.

    “Frameworks such as ToolBench (Qin et al., 2024), WebShop (Yao et al., 2022), and WorkArena (Drouin et al., 2024) rely on fixed APIs or self-hosted environments that abstract away authentication, failure recovery, and long-term user or task state.”

    From A Survey of Agent Memory in the Second Half: Towards Self-Evolving and Long-Horizon Agents (2026) · cites this paper as Drouin et al., 2024

    Section 3.3 (Availability; page unavailable) says WorkArena tasks run on real cloud-based ServiceNow instances, contrary to the claim that WorkArena relies on a self-hosted environment. Section 4.2 (An Ideal Experimental Framework; page unavailable) does say task setup can handle authentication, but it does not establish that WorkArena abstracts away failure recovery or long-term user or task state. (LLM, conf=0.84)

    Checked by a reasoning model — may be wrong.

    Experimental — accuracy not yet measured

  • BrowserGym is the current state-of-the-art method without human-annotated site-specific knowledge.

    “We adopt the current state-of-the-art method without human-annotated site-specific knowledge, BrowserGym (Drouin et al., 2024), which altered the agent default action space.”

    From Agent Workflow Memory (2024) · cites this paper as Drouin et al., 2024

    Section 4 identifies BrowserGym as a browser environment for evaluating web agents, so the citation supports the identification of BrowserGym. Section 4.2 says agents choose their own action space; the paper does not establish BrowserGym as a state-of-the-art method without human-annotated site-specific knowledge or say that it altered an agent's default action space (pages unavailable in our copy). (LLM, conf=0.86)

    Experimental — accuracy not yet measured

  • STeP's accuracy is over 10 percentage points more than the next-best agent.

    “It has an accuracy of 35.8%, more than double the accuracy of the top-performing baseline agent introduced in the WebArena paper, and over 10 percentage points more than the next-best agent [13].”

    From AI Agents That Matter (2024) · cites this paper as [13]

    The Introduction reports a WebArena agent score of 25.4%, which is 10.4 percentage points below the citing sentence’s stated 35.8%. But the source does not report STeP’s 35.8% score or establish that its own agent was the next-best comparator to STeP, so it supports only the comparator score, not the full ranking claim. (LLM, conf=0.84)

    Experimental — accuracy not yet measured

  • AWM achieves the best published results on WebArena, surpassing the BrowserGym baseline by 12.0 absolute points in overall success rate.

    “As shown in Table 1, our AWM achieves the best published results on WebArena, surpassing the BrowserGym baseline by 12 . 0 absolute points and 51 . 1 % relative increase in overall success rate.”

    From Agent Workflow Memory (2024) · cites this paper as Drouin et al., 2024

  • AWM improves over the top published autonomous method by 51.1% relative success rate on WebArena.

    “On WebArena, AWM improves over the top published autonomous method (Drouin et al., 2024) by 51 . 1 %relative success rate, and even outperforms methods augmented with human expert written workflows (Sodhi et al., 2023) by 7 . 9 %.”

    From Agent Workflow Memory (2024) · cites this paper as Drouin et al., 2024

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • WebArena is described as having both 190 and 812 tasks

    Minor

    The related-work section calls WebArena a collection of 190 tasks, but the results table labels the evaluated WebArena benchmark as 812 tasks. The text does not explain a distinction between these two task counts.

  • Zero-success results have positive bootstrap standard errors

    Minor

    With 10 seeds per task, a reported 0.0% success rate across six list-filter tasks implies no successes: even one would give about 1.7%. Bootstrap resampling of those all-zero results would have a standard error of zero, not the positive values shown.

2 of 2

Cites

Flagged first, then by citations

2 of the 32 references couldn’t be matched to a known paper.

Research works printed but not linked: 5

  1. Assouel, R., Marty, T., Caccia, M., Laradji, I., Drouin, A., Rajeswar, S., Palacios, H., Cappart, Q., Vazquez, D., Chapados, N., Gasse, M., and Lacoste, A. The unsolved challenges of LLMs in open-ended web tasks: A case study. In NeurIPS 2023 Foundation Models for Decision Making Workshop, 2023. URL https://openreview.net/forum?id=jt3il4fC5B.
  2. Gur, I., Furuta, H., Huang, A., Safdari, M., Matsuo, Y., Eck, D., and Faust, A. A real-world WebAgent with planning, long context understanding, and program synthesis. arXiv, abs/2307.12856, 2023b. URL https://arxiv.org/abs/2307.12856. · arXiv
  3. Shi, T., Karpathy, A., Fan, L., Hernandez, J., and Liang, P. World of bits: An open-domain platform for web-based agents. In International Conference on Machine Learning (ICML), 2017a.
  4. Shi, T., Karpathy, A., Fan, L., Hernandez, J., and Liang, P. World of bits: An open-domain platform for web-based agents. ICML, 2017b.
  5. Curran Associates, Inc., 2022b. URL https: //proceedings.neurips.cc/paper files/paper/ 2022/file/9d5609613524ecf4f15af0f7b31abca4- Paper-Conference.pdf.

Cited by

Sorted by citations

Related

Authors
Drouin, A., Gasse, M., Caccia, M., Laradji, I. H., Del Verme, M., Marty, T., Boisvert, L., Thakkar, M., Cappart, Q., Vazquez, D., Chapados, N., & Lacoste, A.
Affiliations
  • Mila - Quebec Artificial Intelligence Institute (CA)
  • Polytechnique Montréal (CA)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2024
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2403.07718
arXiv
2403.07718
OpenAlex
W4392781346
Semantic Scholar
fa96417f8766568ba570088513940bbf14e3b356
Cited by
3
References
32
Citations by year
2026: 1 · 2025: 2
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • Digital Access to Libraries (Université catholique de Louvain (UCL), l'Université de Namur (UNamur) and the Université Saint-Louis (USL-B)) (submittedVersion, other-oa)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Engineering › Industrial and Manufacturing Engineering › Robotic Process Automation Applications
  • Physical Sciences › Computer Science › Information Systems › Web Data Mining and Analysis
  • Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
Keywords
LLM agents, web agents, browser automation, ServiceNow, knowledge workers, enterprise software, task automation

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-20.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-20.

Cited sources are checked by gpt-6-sol.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review