WebAgent-R1: Training Web Agents via End-to-End Multi-Turn Reinforcement Learning

“This ensures that the learning process is better aligned with the agent’s most recent actions, often leading to more stable and effective learning (Schulman et al., 2015, 2017).”

“This ensures that the learning process is better aligned with the agent’s most recent actions, often leading to more stable and effective learning (Schulman et al., 2015, 2017).”

“For a comprehensive overview, we refer readers to recent surveys (Wang et al., 2024a; Tseng et al., 2024; Hu et al., 2025; Ning et al., 2025).”

“(Kwon et al., 2023) with a tensor parallel size of 1 and a GPU memory utilization ratio of 0.7.”

“To optimize GPU utilization, we adopt DeepSpeed (Rajbhandari et al., 2020) for distributed training with ZeRO-3 offload, along with gradient checkpointing, FlashAttention-2 (Dao, 2024), and bf16 mixed precision training enabled for computation efficiency.”

“…2024; Abuelsaad et al., 2024; Lutz et al., 2024; Pa-tel et al., 2024; Putta et al., 2024), general computer use (Li et al., 2020; Deng et al., 2023; Yang et al., 2024), and embodied environments (Puig et al., 2018; Shridhar et al., 2020; Toyama et al., 2021; Fan et al., 2022; Huang et al., 2022).”

“However, existing works have primarily focused on single-turn, non-interactive tasks such as mathematical reasoning (Shao et al., 2024; Zeng et al., 2025).”

“…2024; Abuelsaad et al., 2024; Lutz et al., 2024; Pa-tel et al., 2024; Putta et al., 2024), general computer use (Li et al., 2020; Deng et al., 2023; Yang et al., 2024), and embodied environments (Puig et al., 2018; Shridhar et al., 2020; Toyama et al., 2021; Fan et al., 2022; Huang et al., 2022).”

“LLMs have demonstrated promising agentic capabilities, such as breaking down complex tasks into manageable subgoals and reasoning over long horizons (Zhou et al., 2022; Huang et al., 2022; Madaan et al., 2022; Li et al., 2023a,b; Wu et al., 2024; Liu et al., 2025b; Chu et al., 2025).”

“Examples include web-browsing agents [7, 8] that interact with live web content, conversational chatbots [9, 10, 11] that integrate retrieval to enable knowledge-grounded and emotionally aware dialogue, and DeepResearch assistants [12, 13, 14] that autonomously analyze and synthesize information…”

“Reinforcement learning (RL), in contrast, has recently shown strong potential for aligning LLM behavior with high-level objectives, including tool use (Qian et al., 2025; Wang et al., 2025), web navigation (Wei et al., 2025), and search optimization (Jin et al., 2025; Song et al., 2025).”

“…data curation and policy optimization, where agents collect trajectories in executable environments, receive verifiable or environment-derived feedback, and use this feedback to guide task generation, rollout filtering, reward modeling, and multi-turn reinforcement learning [31, 25, 19, 30, 29].”

“…2022; Lee et al., 2024) rely on preference-based rewards, recent trends favor verifiable rewards (RLVR) where correctness is automatically checkable, showing strong potential in mathematics (e.g., GRPO (Shao et al., 2024)) and interactive agent environments (Jimenez et al., 2023; Wei et al., 2025).”

“Prior work has concentrated on improving action selection, with task planning [12, 14, 32], element grounding [49], and model capability [30, 40] all directed toward this goal.”

“RL from online interaction has shown promise for long-horizon interactive tasks (Wei et al., 2025; Qi et al., 2025; Li et al., 2025), though sparse rewards and long trajectories make training challenging.”

“Large Language Models (LLM)-based chatbots and agents have been increasingly popular in mobile agents [43], web agents [48], and personal assistants such as OpenClaw [6].”

“…before offering final answers and actions, have demonstrated great capability and potential in interactive coding (Zhou et al., 2025; Xu et al., 2025; Sun et al., 2025), web-automation (Wei et al., 2025b; Wang et al., 2025a;b), and personalized conversation (Li et al., 2025; Qian et al., 2025b;c).”

“Reinforcement Learning for LLMs. Reinforcement learning has been widely used to align and enhance LLMs [40–42].”

“Earlier work searches over whole agent designs: ADAS programs new agents in code (Hu et al., 2025), AFlow searches over code-represented work-flows (Zhang et al., 2025b), AgentSquare searches a modular design space (Shang et al., 2025), and MaAS samples multi-agent systems from a learned supernet…”

“Recent studies train agents online with multi-turn RL (Qi et al., 2024; Wei et al., 2025; Zhang et al., 2025) using policy-gradient (Feng et al., 2025).”

“WebRL [26] studies curriculum-based online optimization in WebArena, while AgentRL [49], WebAgent-R1 [40], and RAGEN [38] explore multi-turn GRPO-style training and self-evolution in simulated or self-hosted web environments.”

“Studies such as Iterative Reward Calibration demonstrate that dense rewards can misalign policy advantage directions in general agent tasks (Modecrua et al. 2026), while WebAgent-R1 observes scenarios where sparse re-wards outperform dense signals in web navigation (Wei et al. 2025).”

“By leveraging extensive Chain-of-Thought reasoning, LLMs have demonstrated strong capabilities in challenging domains [52, 66] such as mathematical problem solving [12], code generation [65], and agentic workflows [53, 4].”

“Large language models (LLMs) are increasingly deployed in scenarios that demand extremely long contexts, such as repository-level code generation [56, 57], multi-document question answering [17, 45], and multi-turn agentic work-flows [4, 8, 30, 46].”

“Consequently, a growing class of agent RL algorithms has emerged (Feng et al., 2025; Jin et al., 2025; Wei et al., 2025), many of which adapt techniques originally developed for LLM reasoning by maximizing advantage-weighted objectives over trajectories.”

“In the domain of web navigation [25, 27, 99], these agents transcend basic information retrieval by performing sophisticated tasks such as online shopping (comparing prices and checking reviews) [38, 43, 86], booking travel itineraries, or managing digital administrative tasks that require filling…”

“Through text generation, modern LLMs are capable of using external tools such as web browsers [2, 41, 78, 112, 121], email [27, 90], and, more broadly, code-based interfaces [44, 50, 115].”

“…utilization (Qiao et al., 2024; Wang et al., 2025; Yuan et al., 2025; Zhao et al., 2024), embodied planning (Yao et al., 2022b; Wu et al., 2023; Song et al., 2023), web browsing (He et al., 2024; Cai et al., 2025; Wei et al., 2025), and software engineering (Li et al., 2025a; Dong et al., 2025b).”

“Traditional training paradigm for GUI agents primarily relies”

“Examples include web-browsing agents [7, 8] that interact with live web content, conversational chatbots [9, 10, 11] that integrate retrieval to enable knowledge-grounded and emotionally aware dialogue, and DeepResearch assistants [12, 13, 14] that autonomously analyze and synthesize information…”

“Moreover, interaction at inference time is often irreversible: agents cannot rewind actions to correct mistakes or recover from suboptimal decisions.”

“Large language model (LLM) agents have made rapid progress on complex, multi-step tasks such as software engineering (Jimenez et al., 2024; Yang et al., 2024), web navigation (Zhou et al., 2024; He et al., 2024; Wei et al., 2025c), and computer use (Xie et al., 2024; Agashe et al., 2025).”

“Large language model agents are increasingly used for tasks that require sustained interaction with ex-ternal tools and environments, including web navigation (Wei et al., 2025), household tasks (Shrid-har et al., 2020), and search-augmented reasoning (Huang et al., 2026).”

“Building on this paradigm, agentic RL extends it to multi-turn agent interactions, typically rewarding the correctness of the final answer or task success (Jin et al., 2025; Zheng et al., 2025; Wei et al., 2025).”

“Secondly, the outcome reward is a single value per episode, far too sparse to shape multiple structured decisions with reinforcement learning (RL) (Shao et al., 2024; Wei et al., 2025).”

“…gyms make evaluation or training reproducible on one platform: BrowserGym unifies web benchmarks behind one interface, WebGym, MobileGym, and SCALECUA run online RL on hundreds of parallel environments, and ClawGUI provides a framework for training, evaluating, and deploying GUI agents [14–18].”

“Critic-free RL like GRPO (Shao et al., 2024) is widely adopted for LLM agents (Qian et al., 2025; Jin et al., 2025; Wei et al., 2025), but suffers from sparse-reward advantage vanishment (Yu et al., 2025) and coarse trajectory-level credit assignment (Tan et al., 2026).”

“We use WebAgent-R1 and MemAgent as representative methods from these respective categories, re-implementing both baselines with reference to their official implementations.”

“Following the success of GRPO for reasoning models [8, 9], it has been applied with GRPO, PPO or their variants to search [3], tool use [4, 5], web navigation [6], and games and embodied text environments [7, 49].”

“ReAct alternates reasoning and actions, NNetNav generates additional interaction data, and recent methods such as WebRL and WebAgent R1 apply reinforcement learning to long browser tasks [Yao et al., 2023, Murty et al., 2025, Qi et al., 2025, Wei et al., 2025].”

“More recent work has extended this paradigm to broader verifiable settings, including formal reasoning Shao et al. (2024), retrieval augmented reasoning Jin et al. (2025), and agentic tasks with environment feedback Wei et al. (2025).”

“Search-R1 [Jin et al., 2025] trains models to interleave reasoning with search through RL, while WebAgent-R1 [Wei et al., 2025] trains web agents through multi-turn environment interactions.”

“…LLMs have been studied in embodied environments, web navigation, tool-use tasks, and application-centered workflows, where success requires a sequence of environment-dependent actions rather than a single response (Shridhar et al., 2020; Yao et al., 2022a; Trivedi et al., 2024; Wei et al., 2025).”

“This paradigm spans web navigation, information seeking, and tool use (Wei et al., 2025; Jin et al., 2025; Chen et al., 2025; Qian et al., 2026).”

“P2A instead ties within-task memory to the decision loop: only probed, acted-upon, or explicitly committed evidence is retained, while long-horizon RL remains an orthogonal direction (Wei et al., 2025; Ye et al., 2025).”

“Reward modeling for web agents initially focused on trajectory-level Outcome Reward Models (ORMs), such as the outcome evaluators in WebRL (Qi et al., 2025) and WebAgent-R1 (Wei et al., 2025), which assess overall task completion to guide reinforcement learning.”

“WebAgent-R1 further develops outcome-reward training for web agents by integrating asynchronous online exploration with reasoning-oriented warm-up in an end-to-end multi-turn RL framework (Wei et al., 2025).”

“WebRL and WebAgent-R1 train in live environments on task success (Qi et al., 2025; Wei et al., 2025), while UI-R1 and GUI-R1 apply group relative policy optimization with verifiable action rewards (Lu et al., 2026; Luo et al., 2025), building on GRPO (Shao et al., 2024) and its refinements (Liu et…”

“The same approach is now applied to LLM agents in sequential decision tasks such as web navigation, embodied control, and tool use (Zeng et al., 2024; Song et al., 2024; Wei et al., 2025).”

“Such progress has led to growing optimism that language agents can serve as a general interface for web automation (He et al. 2024; Wei et al. 2025) and other tasks in diverse interactive environments (Wang et al. 2025; He et al. 2025).”

“Similar ideas appeared in web navigation (Wei et al. 2025) and environment-free training (Li et al. 2025d).”

“Reasoning has also helped in interactive domains such as coding and web agents [10, 60], where models call tools, observe feedback, and revise their behavior online [42, 47, 51].”

“…agent perceives a web page, decides where to act, and carries the goal across many steps of clicking, typing, and navigating until the task is done (Wei et al., 2025; Xiao et al., 2026; Logeswaran et al., 2026; Gur et al., 2024; Deng et al., 2023; Zhou et al., 2023; He et al., 2024a; Wu et al.,…”

“Specifically, Group Relative Policy Optimization (GRPO) [13] has been demonstrated as an effective algorithm and is widely applied to improve agent performance across various tasks [9, 43, 44, 45].”

“Beyond imitation, AutoWebGLM bootstraps with rejection sampling and RL (Lai et al. 2024), and WebAgent-R1 applies end-to-end multi-turn RL (Wei et al. 2025); data-centric RL systems are summarized above.”

“This architecture has demonstrated strong zero-shot and few-shot performance across complex domains such as web navigation, multi-step mathematical reasoning, and automated software engineering [11, 16, 23].”

“This follows recent multi-turn agent RL systems that move beyond static imitation toward interaction-based improvement [81, 84, 117].”

“(2) Online, multi-turn and self-evolving RL frameworks leveraging reward model and interactive environments, such as ZeroGUI [45], WebEvolver [9], MobileGUI-RL [36], SEAgent [37] and WebAgent-R1 [39].”

“…that improved optimization does not automatically yield OOD robustness (Chu et al., 2025; Liu et al., 2025); models may continue to rely on brittle heuristics inherited from the training distribution (Chu et al., 2025; Zhang et al., 2025; Zhou et al., 2024; Chen et al., 2025; Wei et al., 2025).”

“( 6) Local session memory is updated after each wave; at the end of the sample, the trajectory is distilled into procedural rules and declarative facts that update the persistent memory stores.”

“…attempts to improve these include more aggressive filtering of the DOM or accessibility tree to feed the LM (Schiepanski & Pi¨el, 2025) and reinforcement-learning to train smaller models for generating actions given web browser state (Wang et al., 2025; Wei et al., 2025; Bhathal & Gupta, 2025).”

“However, both online RL [13], [14] and offline RL [15], [16] face significant data-centric challenges.”

“…delayed rewards have been tackled by reinforcement learning (RL) in the context of multi-turn conversations with human users, e.g., Zhou et al. (2024; 2025); Jaques et al. (2020); Jang et al. (2022), as well as in the context of web-agents Wei et al. (2025); Chae et al. (2025); He et al. (2024).”

“Other self-improving frameworks, including WebRL [27], WebAgent-R1 [28], and Agent Q [29], improve agent behavior through online exploration, reinforcement learning, or search-based correction.”

“Our theoretical analysis shows that DFPO’s policy gradient can be decomposed into the M-GRPO (Shao et al., 2024; Wei et al., 2025) gradient plus an implicit bias term induced by DFPO’s draft–solution structure, and under certain conditions, is guaranteed to assign an advantage bonus to optimal…”

“While such agents can solve non-trivial tasks in embodied-like text environments and web navigation tasks (Shridhar et al., 2021; He et al., 2024; Wei et al., 2025), improving them with end-to-end reinforcement learning remains challenging, especially in long-horizon settings with sparse and…”

“Given that the full evaluation set in WebArena (Zhou et al., 2024) is lengthy and includes many similar tasks, we follow prior work (Qi et al., 2025; Wei et al., 2025) and evaluate our trained agents on WebArena-Lite (Liu et al., 2025), a more efficient and balanced subset of 165 high-quality,…”