The DOI matches its DataCite record; 57 of the 64 references it cites were found in a registry or an index.
WebAgent-R1: Training Web Agents via End-to-End Multi-Turn Reinforcement Learning
Open accessAbstract
While reinforcement learning (RL) has demonstrated remarkable success in enhancing large language models (LLMs), it has primarily focused on single-turn tasks such as solving math problems. Training effective web agents for multi-turn interactions remains challenging due to the complexity of long-horizon decision-making across dynamic web interfaces. In this work, we present WebAgent-R1, a simple yet effective end-to-end multi-turn RL framework for training web agents. It learns directly from online interactions with web environments by asynchronously generating diverse trajectories, entirely guided by binary rewards depending on task success. Experiments on the WebArena-Lite benchmark demonstrate the effectiveness of WebAgent-R1, boosting the task success rate of Qwen-2.5-3B from 6.1% to 33.9% and Llama-3.1-8B from 8.5% to 44.8%, significantly outperforming existing state-of-the-art methods and strong proprietary models such as OpenAI o3. In-depth analyses reveal the effectiveness of the thinking-based prompting strategy and test-time scaling through increased interactions for web tasks. We further investigate different RL initialization policies by introducing two variants, namely WebAgent-R1-Zero and WebAgent-R1-CoT, which highlight the importance of the warm-up training stage (i.e., behavior cloning) and provide insights on incorporating long chain-of-thought (CoT) reasoning in web agents.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“Reinforcement learning (RL), in contrast, has recently shown strong potential for aligning LLM behavior with high-level objectives, including tool use (Qian et al., 2025; Wang et al., 2025), web navigation (Wei et al., 2025), and search optimization (Jin et al., 2025; Song et al., 2025).”
From Memory-R1: Enhancing Large Language Model Agents to Manage and Utilize Memories via Reinforcement Learning (2025) · cites this paper as Wei et al., 2025
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Proximal Policy Optimization Algorithms
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., & Klimov, O. (2017) · arXiv · doi:10.48550/arxiv.1707.06347 · Cited by 10,625 · cited 1×
“This ensures that the learning process is better aligned with the agent’s most recent actions, often leading to more stable and effective learning (Schulman et al., 2015, 2017).”
High-Dimensional Continuous Control Using Generalized Advantage Estimation
Schulman, J., Moritz, P., Levine, S., Jordan, M., & Abbeel, P. (2015) · arXiv · doi:10.48550/arxiv.1506.02438 · Cited by 2,103 · cited 1×
“This ensures that the learning process is better aligned with the agent’s most recent actions, often leading to more stable and effective learning (Schulman et al., 2015, 2017).”
A survey on large language model based autonomous agents
Wang, L., Ma, C., Feng, X., Zhang, Z., Yang, H., Zhang, J., Chen, Z., Tang, J., Chen, X., Lin, Y., et al. (2024) · Frontiers of Computer Science · doi:10.1007/s11704-024-40231-1 · Cited by 1,941 · cited 1×
“For a comprehensive overview, we refer readers to recent surveys (Wang et al., 2024a; Tseng et al., 2024; Hu et al., 2025; Ning et al., 2025).”
Self-citationEfficient Memory Management for Large Language Model Serving with PagedAttention
Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez, J., Zhang, H., & Stoica, I. (2023) · Proceedings of the 29th Symposium on Operating Systems Principles · doi:10.1145/3600006.3613165 · Cited by 1,817 · cited 1×
“(Kwon et al., 2023) with a tensor parallel size of 1 and a GPU memory utilization ratio of 0.7.”
InfluentialSelf-citation(2020) · doi:10.1109/sc41405.2020.00024 · Cited by 928 · cited 1×
“To optimize GPU utilization, we adopt DeepSpeed (Rajbhandari et al., 2020) for distributed training with ZeRO-3 offload, along with gradient checkpointing, FlashAttention-2 (Dao, 2024), and bf16 mixed precision training enabled for computation efficiency.”
ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Tasks
Shridhar, M., Thomason, J., Gordon, D., Bisk, Y., Han, W., Mottaghi, R., Zettlemoyer, L., & Fox, D. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.01075 · Cited by 540 · cited 1×
“…2024; Abuelsaad et al., 2024; Lutz et al., 2024; Pa-tel et al., 2024; Putta et al., 2024), general computer use (Li et al., 2020; Deng et al., 2023; Yang et al., 2024), and embodied environments (Puig et al., 2018; Shridhar et al., 2020; Toyama et al., 2021; Fan et al., 2022; Huang et al., 2022).”
SimpleRL-Zoo: Investigating and Taming Zero Reinforcement Learning for Open Base Models in the WildPreprint
Zeng, W., Huang, Y., Liu, Q., Liu, W., He, K., Ma, Z., & He, J. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2503.18892 · Cited by 463 · cited 1×
“However, existing works have primarily focused on single-turn, non-interactive tasks such as mathematical reasoning (Shao et al., 2024; Zeng et al., 2025).”
RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement LearningPreprint
Wang, Z., Wang, K., Wang, Q., Zhang, P., Li, L., Yang, Z., Jin, X., Yu, K. F., Nguyen, M. N., Liu, L., et al. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2504.20073 · Cited by 443
Self-citationVirtualHome: Simulating Household Activities Via Programs
Puig, X., Ra, K., Boben, M., Li, J., Wang, T., Fidler, S., & Torralba, A. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00886 · Cited by 378 · cited 1×
“…2024; Abuelsaad et al., 2024; Lutz et al., 2024; Pa-tel et al., 2024; Putta et al., 2024), general computer use (Li et al., 2020; Deng et al., 2023; Yang et al., 2024), and embodied environments (Puig et al., 2018; Shridhar et al., 2020; Toyama et al., 2021; Fan et al., 2022; Huang et al., 2022).”
Camel: Communicative agents for" mind" exploration of large language model society
Li, G., Hammoud, H., Itani, H., Khizbullin, D., & Ghanem, B. (2023) · Advances in Neural Information Processing Systems 36 · doi:10.52202/075280-2264 · Cited by 314 · cited 1×
“LLMs have demonstrated promising agentic capabilities, such as breaking down complex tasks into manageable subgoals and reasoning over long horizons (Zhou et al., 2022; Huang et al., 2022; Madaan et al., 2022; Li et al., 2023a,b; Wu et al., 2024; Liu et al., 2025b; Chu et al., 2025).”
Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agents
Huang, W., Abbeel, P., Pathak, D., & Mordatch, I. (2022) · arXiv · doi:10.48550/arxiv.2201.07207 · Cited by 304 · cited 2×
“…2024; Abuelsaad et al., 2024; Lutz et al., 2024; Pa-tel et al., 2024; Putta et al., 2024), general computer use (Li et al., 2020; Deng et al., 2023; Yang et al., 2024), and embodied environments (Puig et al., 2018; Shridhar et al., 2020; Toyama et al., 2021; Fan et al., 2022; Huang et al., 2022).”
Language Models of Code are Few-Shot Commonsense Learners
Madaan, A., Zhou, S., Alon, U., Yang, Y.-M., & Neubig, G. (2022) · Conference on Empirical Methods in Natural Language Processing · doi:10.48550/arxiv.2210.07128 · Cited by 285 · cited 1×
“LLMs have demonstrated promising agentic capabilities, such as breaking down complex tasks into manageable subgoals and reasoning over long horizons (Zhou et al., 2022; Huang et al., 2022; Madaan et al., 2022; Li et al., 2023a,b; Wu et al., 2024; Liu et al., 2025b; Chu et al., 2025).”
ZeroSearch: Incentivize the Search Capability of LLMs without SearchingPreprint
Sun, H., Qiao, Z., Guo, J., Fan, X., Hou, Y., Jiang, Y., Xie, P., Zhang, Y., Huang, F., & Zhou, J. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2505.04588 · Cited by 232 · cited 1×
“Recent efforts have made some progress in this direction, such as training LLM agents to repeatedly use search engines (Jin et al., 2025; Sun et al., 2025; Chen et al., 2025; Song et al., 2025; Wang et al., 2025a), but typically constrain actions to simple API calls without real environment…”
Mapping Natural Language Instructions to Mobile UI Action Sequences
Li, Y., He, J., Zhou, X., Zhang, Y., & Baldridge, J. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.729 · Cited by 209 · cited 1×
“…2022; Ma et al., 2023; Gur et al., 2024; Abuelsaad et al., 2024; Lutz et al., 2024; Pa-tel et al., 2024; Putta et al., 2024), general computer use (Li et al., 2020; Deng et al., 2023; Yang et al., 2024), and embodied environments (Puig et al., 2018; Shridhar et al., 2020; Toyama et al., 2021; Fan…”
WebRL: Training LLM Web Agents via Self-Evolving Online Curriculum Reinforcement LearningPreprint
Qi, Z., Liu, X., Iong, I. L., Lai, H., Sun, X., Zhao, W., Yu, Y., Yang, X., Sun, J., Yao, S., et al. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2411.02337 · Cited by 199 · cited 18×
“However, most of this line of research has heavily relied on offline or iterative off-policy RL solutions (Peng et al., 2019; Pan et al., 2024; Qi et al., 2025), which break the end-to-end interaction between the web agent and environment, and introduce additional complexities such as trajectory…”
InfluentialThe Surprising Effectiveness of Negative Reinforcement in LLM Reasoning
Zhu, X., Xia, M., Wei, Z., et al. (2025) · Neural Information Processing Systems · doi:10.48550/arxiv.2506.01347 · Cited by 181
Self-citationAdvantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement LearningPreprint
Peng, X. B., Kumar, A., Zhang, G., & Levine, S. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1910.00177 · Cited by 158 · cited 2×
“However, most of this line of research has heavily relied on offline or iterative off-policy RL solutions (Peng et al., 2019; Pan et al., 2024; Qi et al., 2025), which break the end-to-end interaction between the web agent and environment, and introduce additional complexities such as trajectory…”
SWEET-RL: Training Multi-Turn LLM Agents on Collaborative Reasoning TasksPreprint
Zhou, Y., Jiang, S., Tian, Y., Weston, J., Levine, S., Sukhbaatar, S., & Li, X. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2503.15478 · Cited by 154
AutoWebGLM: A Large Language Model-based Web Navigating Agent
Lai, H., Liu, X., Iong, I. L., Yao, S., Chen, Y., Shen, P., Yu, H., Zhang, H., Zhang, X., Dong, Y., et al. (2024) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3637528.3671620 · Cited by 146 · cited 1×
“…methods (Wang et al., 2024b; Sodhi et al., 2024; Fu et al., 2024; Zhang et al., 2025; Yang et al., 2025b) or behavior cloning (BC), which imi-tates demonstrated trajectories via supervised fine-tuning (Yin et al., 2024; Hong et al., 2024; Lai et al., 2024; He et al., 2024b; Putta et al., 2024).”
FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning
Dao, T. (2023) · arXiv · doi:10.48550/arxiv.2307.08691 · Cited by 143
Mind2Web: Towards a Generalist Agent for the Web
Deng, X., Gu, Y., Zheng, B., Chen, S., Stevens, S., Wang, B., Sun, H., & Su, Y. (2023) · arXiv · doi:10.48550/arxiv.2306.06070 · Cited by 134 · cited 2×
“Following prior work (Qi et al., 2025), we adopt WebArena (Zhou et al., 2024a) as the web environment over other simulated or static environments such as WebShop (Yao et al., 2022) or Mind2Web (Deng et al., 2023) for greater practicality—”
InfluentialPix2Struct: Screenshot Parsing as Pretraining for Visual Language UnderstandingPreprint
Lee, K., Joshi, M., Turc, I., Hu, H., Liu, F., Eisenschlos, J. M., Khandelwal, U., Shaw, P., Chang, M.-W., & Toutanova, K. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2210.03347 · Cited by 127 · cited 1×
“Complementary to this line of work, GUI agents leverage additional multimodal inputs such as screenshots, enabling visual-guided interactions with the environment (Lee et al., 2023; Shaw et al., 2023; Zheng et al., 2024; He et al., 2024a,b; Koh et al., 2024; Kil et al., 2024; Lei et al., 2025; Liu…”
Zhang, Y., Ma, Z.-J., Ma, Y., et al. (2024) · AAAI Conference on Artificial Intelligence · doi:10.48550/arxiv.2408.15978 · Cited by 127 · cited 1×
“Early works on web agents primarily relied on prompting-based methods (Wang et al., 2024b; Sodhi et al., 2024; Fu et al., 2024; Zhang et al., 2025; Yang et al., 2025b) or behavior cloning (BC), which imi-tates demonstrated trajectories via supervised fine-tuning (Yin et al., 2024; Hong et al.,…”
WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agents
Yao, S., Chen, H., Yang, J., & Narasimhan, K. (2022) · arXiv · doi:10.48550/arxiv.2207.01206 · Cited by 107 · cited 2×
“Following prior work (Qi et al., 2025), we adopt WebArena (Zhou et al., 2024a) as the web environment over other simulated or static environments such as WebShop (Yao et al., 2022) or Mind2Web (Deng et al., 2023) for greater practicality—”
InfluentialAndroidEnv: A Reinforcement Learning Platform for AndroidPreprint
Toyama, D., Hamel, P., Gergely, A., Comanici, G., Glaese, A., Ahmed, Z., Jackson, T., Mourad, S., & Precup, D. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2105.13231 · Cited by 106 · cited 1×
“…2024; Abuelsaad et al., 2024; Lutz et al., 2024; Pa-tel et al., 2024; Putta et al., 2024), general computer use (Li et al., 2020; Deng et al., 2023; Yang et al., 2024), and embodied environments (Puig et al., 2018; Shridhar et al., 2020; Toyama et al., 2021; Fan et al., 2022; Huang et al., 2022).”
VisualAgentBench: Towards Large Multimodal Models as Visual Foundation AgentsPreprint
Liu, X., Zhang, T., Gu, Y., Iong, I. L., Xu, Y., Song, X., Zhang, S., Lai, H., Liu, X., Zhao, H., et al. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2408.06327 · Cited by 100 · cited 12×
“3 Experiments”
InfluentialFu, Y., Kim, D.-K., Kim, J., Sohn, S., Logeswaran, L., Bae, K., & Lee, H. (2024) · Neural Information Processing Systems · doi:10.48550/arxiv.2403.08978 · Cited by 98 · cited 1×
“Early works on web agents primarily relied on prompting-based methods (Wang et al., 2024b; Sodhi et al., 2024; Fu et al., 2024; Zhang et al., 2025; Yang et al., 2025b) or behavior cloning (BC), which imi-tates demonstrated trajectories via supervised fine-tuning (Yin et al., 2024; Hong et al.,…”
Two Tales of Persona in LLMs: A Survey of Role-Playing and Personalization
Tseng, Y.-M., Huang, Y.-C., Hsiao, T.-Y., Chen, W.-L., Huang, C.-W., Meng, Y., & Chen, Y.-N. (2024) · arXiv · doi:10.48550/arxiv.2406.01171 · Cited by 87
LASER: LLM Agent with State-Space Exploration for Web Navigation
Ma, K., Zhang, H., Wang, H., Pan, X., & Yu, D. (2023) · arXiv.org · doi:10.48550/arxiv.2309.08172 · Cited by 73 · cited 1×
“…LLM-based agents have been applied to a variety of real-world interactive tasks, including web navigation (Nakano et al., 2021; Yao et al., 2022; Ma et al., 2023; Gur et al., 2024; Abuelsaad et al., 2024; Lutz et al., 2024; Pa-tel et al., 2024; Putta et al., 2024), general computer use (Li et…”
SWE-bench: Can Language Models Resolve Real-World GitHub Issues?
Jimenez, C. E., Yang, J., Wettig, A., Yao, S., Pei, K., Press, O., & Narasimhan, K. (2023) · arXiv · doi:10.48550/arxiv.2310.06770 · Cited by 72 · cited 2×
“…2024; Abuelsaad et al., 2024; Lutz et al., 2024; Pa-tel et al., 2024; Putta et al., 2024), general computer use (Li et al., 2020; Deng et al., 2023; Yang et al., 2024), and embodied environments (Puig et al., 2018; Shridhar et al., 2020; Toyama et al., 2021; Fan et al., 2022; Huang et al., 2022).”
MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledge
Fan, L., Wang, G., Jiang, Y., Mandlekar, A., Yang, Y., Zhu, H., Tang, A., Huang, D.-A., Zhu, Y., & Anandkumar, A. (2022) · arXiv · doi:10.48550/arxiv.2206.08853 · Cited by 72 · cited 1×
“…2024; Abuelsaad et al., 2024; Lutz et al., 2024; Pa-tel et al., 2024; Putta et al., 2024), general computer use (Li et al., 2020; Deng et al., 2023; Yang et al., 2024), and embodied environments (Puig et al., 2018; Shridhar et al., 2020; Toyama et al., 2021; Fan et al., 2022; Huang et al., 2022).”
Web Agents with World Models: Learning and Leveraging Environment Dynamics in Web NavigationPreprint
Chae, H., Kim, N., Ong, K. T.-I., Gwak, M., Song, G., Kim, J., Kim, S., Lee, D., & Yeo, J. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2410.13232 · Cited by 69 · cited 1×
“Their effectiveness in multi-turn, interactive environments—particularly in complex scenarios requiring long-horizon decision-making and domain-specific skills, such as web browsing (Zhou et al., 2024a; He et al., 2024a; Chae et al., 2025)— still remains underexplored.”
He, H., Yao, W., Ma, K.-X., et al. (2024) · Annual Meeting of the Association for Computational Linguistics · doi:10.48550/arxiv.2410.19609 · Cited by 50 · cited 1×
“…methods (Wang et al., 2024b; Sodhi et al., 2024; Fu et al., 2024; Zhang et al., 2025; Yang et al., 2025b) or behavior cloning (BC), which imi-tates demonstrated trajectories via supervised fine-tuning (Yin et al., 2024; Hong et al., 2024; Lai et al., 2024; He et al., 2024b; Putta et al., 2024).”
Self-citationScaffolding Coordinates to Promote Vision-Language Coordination in Large Multi-Modal ModelsPreprint
Lei, X., Yang, Z., Chen, X., Li, P., & Liu, Y. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2402.12058 · Cited by 41 · cited 1×
“…this line of work, GUI agents leverage additional multimodal inputs such as screenshots, enabling visual-guided interactions with the environment (Lee et al., 2023; Shaw et al., 2023; Zheng et al., 2024; He et al., 2024a,b; Koh et al., 2024; Kil et al., 2024; Lei et al., 2025; Liu et al., 2025a).”
Self-citationShow Me More Details: Discovering Hierarchies of Procedures from Semi-structured Web Data
Zhou, S., Zhang, L., Yang, Y., et al. (2022) · Annual Meeting of the Association for Computational Linguistics · doi:10.48550/arxiv.2203.07264 · Cited by 33 · cited 1×
“LLMs have demonstrated promising agentic capabilities, such as breaking down complex tasks into manageable subgoals and reasoning over long horizons (Zhou et al., 2022; Huang et al., 2022; Madaan et al., 2022; Li et al., 2023a,b; Wu et al., 2024; Liu et al., 2025b; Chu et al., 2025).”
WebArena: A Realistic Web Environment for Building Autonomous Agents
Zhou, S., Xu, F. F., Zhu, H., Zhou, X., Lo, R., Sridhar, A., Cheng, X., Ou, T., Bisk, Y., Fried, D., et al. (2023) · arXiv · doi:10.48550/arxiv.2307.13854 · Cited by 31 · cited 16×
“Empirically, W EB A GENT -R1 achieves new state-of-the-art re-sults on the WebArena-Lite benchmark.”
InfluentialSWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering
Yang, J., Jimenez, C. E., Wettig, A., Lieret, K., Yao, S., Narasimhan, K., & Press, O. (2024) · arXiv · doi:10.48550/arxiv.2405.15793 · Cited by 31
Agent-E: From Autonomous Web Navigation to Foundational Design Principles in Agentic Systems
Abuelsaad, T., Akkil, D., Dey, P., Jagmohan, A., Vempaty, A., & Kokku, R. (2024) · arXiv · doi:10.48550/arxiv.2407.13032 · Cited by 31 · cited 1×
“…to a variety of real-world interactive tasks, including web navigation (Nakano et al., 2021; Yao et al., 2022; Ma et al., 2023; Gur et al., 2024; Abuelsaad et al., 2024; Lutz et al., 2024; Pa-tel et al., 2024; Putta et al., 2024), general computer use (Li et al., 2020; Deng et al., 2023; Yang…”
SelfElicit: Your Language Model Secretly Knows Where is the Relevant Evidence
Liu, Z., Amjad, R. A., Adkathimar, R., et al. (2025) · Annual Meeting of the Association for Computational Linguistics · doi:10.48550/arxiv.2502.08767 · Cited by 30 · cited 1×
“LLMs have demonstrated promising agentic capabilities, such as breaking down complex tasks into manageable subgoals and reasoning over long horizons (Zhou et al., 2022; Huang et al., 2022; Madaan et al., 2022; Li et al., 2023a,b; Wu et al., 2024; Liu et al., 2025b; Chu et al., 2025).”
WILBUR: Adaptive In-Context Learning for Robust and Accurate Web AgentsPreprint
Lutz, M., Bohra, A., Saroyan, M., Harutyunyan, A., & Campagna, G. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2404.05902 · Cited by 26 · cited 1×
“…interactive tasks, including web navigation (Nakano et al., 2021; Yao et al., 2022; Ma et al., 2023; Gur et al., 2024; Abuelsaad et al., 2024; Lutz et al., 2024; Pa-tel et al., 2024; Putta et al., 2024), general computer use (Li et al., 2020; Deng et al., 2023; Yang et al., 2024), and…”
OS Agents: A Survey on MLLM-based Agents for Computer, Phone and Browser Use
Hu, X., Tao, X., Yi, B., Wei, Z., Xiao, R., Chen, Y., Ye, J., Tao, M., Zhou, X., Zhao, Z., et al. (2025) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2025.acl-long.369 · Cited by 23 · cited 2×
“For a comprehensive overview, we refer readers to recent surveys (Wang et al., 2024a; Tseng et al., 2024; Hu et al., 2025; Ning et al., 2025).”
Self-citationDigiRL: Training In-The-Wild Device-Control Agents with Autonomous Reinforcement Learning
Bai, H., Zhou, Y., Cemri, M., Pan, J., Suhr, A., Levine, S., & Kumar, A. (2024) · arXiv · doi:10.48550/arxiv.2406.11896 · Cited by 20 · cited 3×
“…2024; Qi et al., 2025), which break the end-to-end interaction between the web agent and environment, and introduce additional complexities such as trajectory filtering (Bai et al., 2024), outcome reward model training (Qi et al., 2025), or iterative optimization procedures (Zhou et al., 2024b).”
ArCHer: Training Language Model Agents via Hierarchical Multi-Turn RLPreprint
Zhou, Y., Zanette, A., Pan, J., Levine, S., & Kumar, A. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2402.19446 · Cited by 18 · cited 2×
“…2024; Qi et al., 2025), which break the end-to-end interaction between the web agent and environment, and introduce additional complexities such as trajectory filtering (Bai et al., 2024), outcome reward model training (Qi et al., 2025), or iterative optimization procedures (Zhou et al., 2024b).”
A Real-World WebAgent with Planning, Long Context Understanding, and Program Synthesis
Gur, I., Furuta, H., Huang, A., Safdari, M., Matsuo, Y., Eck, D., & Faust, A. (2023) · arXiv · doi:10.48550/arxiv.2307.12856 · Cited by 17 · cited 1×
“…have been applied to a variety of real-world interactive tasks, including web navigation (Nakano et al., 2021; Yao et al., 2022; Ma et al., 2023; Gur et al., 2024; Abuelsaad et al., 2024; Lutz et al., 2024; Pa-tel et al., 2024; Putta et al., 2024), general computer use (Li et al., 2020; Deng et…”
Agent Lumos: Unified and Modular Training for Open-Source Language Agents
Yin, D., Brahman, F., Ravichander, A., Chandu, K., Chang, K.-W., Choi, Y., & Lin, B. Y. (2024) · Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2024.acl-long.670 · Cited by 17 · cited 1×
“…methods (Wang et al., 2024b; Sodhi et al., 2024; Fu et al., 2024; Zhang et al., 2025; Yang et al., 2025b) or behavior cloning (BC), which imi-tates demonstrated trajectories via supervised fine-tuning (Yin et al., 2024; Hong et al., 2024; Lai et al., 2024; He et al., 2024b; Putta et al., 2024).”
Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning
Jin, B., Zeng, H., Yue, Z., Yoon, J., Arik, S., Wang, D., Zamani, H., & Han, J. (2025) · arXiv · doi:10.48550/arxiv.2503.09516 · Cited by 13 · cited 1×
“Recent efforts have made some progress in this direction, such as training LLM agents to repeatedly use search engines (Jin et al., 2025; Sun et al., 2025; Chen et al., 2025; Song et al., 2025; Wang et al., 2025a), but typically constrain actions to simple API calls without real environment…”
Dual-View Visual Contextualization for Web Navigation
Kil, J., Song, C. H., Zheng, B., Deng, X., Su, Y., & Chao, W. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01369 · Cited by 12 · cited 1×
“…this line of work, GUI agents leverage additional multimodal inputs such as screenshots, enabling visual-guided interactions with the environment (Lee et al., 2023; Shaw et al., 2023; Zheng et al., 2024; He et al., 2024a,b; Koh et al., 2024; Kil et al., 2024; Lei et al., 2025; Liu et al., 2025a).”
GPT-4V(ision) is a Generalist Web Agent, if Grounded
Zheng, B., Gou, B., Kil, J., Sun, H., & Su, Y. (2024) · arXiv · doi:10.48550/arxiv.2401.01614 · Cited by 10 · cited 1×
“…this line of work, GUI agents leverage additional multimodal inputs such as screenshots, enabling visual-guided interactions with the environment (Lee et al., 2023; Shaw et al., 2023; Zheng et al., 2024; He et al., 2024a,b; Koh et al., 2024; Kil et al., 2024; Lei et al., 2025; Liu et al., 2025a).”
Agent Q: Advanced Reasoning and Learning for Autonomous AI Agents
Putta, P., Mills, E., Garg, N., Motwani, S., Finn, C., Garg, D., & Rafailov, R. (2024) · arXiv · doi:10.48550/arxiv.2408.07199 · Cited by 10 · cited 2×
“…(Nakano et al., 2021; Yao et al., 2022; Ma et al., 2023; Gur et al., 2024; Abuelsaad et al., 2024; Lutz et al., 2024; Pa-tel et al., 2024; Putta et al., 2024), general computer use (Li et al., 2020; Deng et al., 2023; Yang et al., 2024), and embodied environments (Puig et al., 2018;…”
RAST: Reasoning Activation in LLMs via Small-model Transfer
Ouyang, S., Zhu, X.-Y., Xiao, Z.-L., et al. (2025) · Neural Information Processing Systems · doi:10.48550/arxiv.2506.15710 · Cited by 10 · cited 1×
“However, most prior work focuses on single-turn tasks such as math problems (Shao et al., 2024; Zhu et al., 2025; Shao et al., 2025; Ouyang et al., 2025; Wei et al., 2025), with lim-8 ited exploration in multi-turn settings (Zhou et al., 2024b, 2025).”
Take a Break in the Middle: Investigating Subgoals towards Hierarchical Script Generation
Li, X., Cao, Y., Chen, M., et al. (2023) · Annual Meeting of the Association for Computational Linguistics · doi:10.48550/arxiv.2305.10907 · Cited by 8
Beyond Outcome Reward: Decoupling Search and Answering Improves LLM Agents
Wang, Y., Wei, Z., Zhu, X.-Y., et al. (2025) · arXiv.org · doi:10.48550/arxiv.2510.04695 · Cited by 6 · cited 1×
“…efforts have made some progress in this direction, such as training LLM agents to repeatedly use search engines (Jin et al., 2025; Sun et al., 2025; Chen et al., 2025; Song et al., 2025; Wang et al., 2025a), but typically constrain actions to simple API calls without real environment interaction.”
Self-citationAutonomous Evaluation and Refinement of Digital Agents
Pan, J., Zhang, Y., Tomlin, N., Zhou, Y., Levine, S., & Suhr, A. (2024) · arXiv · doi:10.48550/arxiv.2404.06474 · Cited by 3 · cited 1×
“However, most of this line of research has heavily relied on offline or iterative off-policy RL solutions (Peng et al., 2019; Pan et al., 2024; Qi et al., 2025), which break the end-to-end interaction between the web agent and environment, and introduce additional complexities such as trajectory…”
VisualWebArena: Evaluating Multimodal Agents on Realistic Visual Web Tasks
Koh, J. Y., Lo, R., Jang, L., Duvvur, V., Lim, M. C., Huang, P.-Y., Neubig, G., Zhou, S., Salakhutdinov, R., & Fried, D. (2024) · arXiv · doi:10.48550/arxiv.2401.13649 · Cited by 1 · cited 1×
“…this line of work, GUI agents leverage additional multimodal inputs such as screenshots, enabling visual-guided interactions with the environment (Lee et al., 2023; Shaw et al., 2023; Zheng et al., 2024; He et al., 2024a,b; Koh et al., 2024; Kil et al., 2024; Lei et al., 2025; Liu et al., 2025a).”
Spurious Rewards: Rethinking Training Signals in RLVR
Shao, R., Li, S. S., Xin, R., Geng, S., Wang, Y., Oh, S., Du, S. S., Lambert, N., Min, S., Krishna, R., et al. (2025) · arXiv · doi:10.48550/arxiv.2506.10947 · Cited by 1 · cited 1×
“However, most prior work focuses on single-turn tasks such as math problems (Shao et al., 2024; Zhu et al., 2025; Shao et al., 2025; Ouyang et al., 2025; Wei et al., 2025), with lim-8 ited exploration in multi-turn settings (Zhou et al., 2024b, 2025).”
SteP: Stacked LLM Policies for Web Actions
Sodhi, P., Branavan, S. R. K., Artzi, Y., & McDonald, R. (2023) · arXiv · doi:10.48550/arxiv.2310.03720 · Cited by 1 · cited 1×
“Early works on web agents primarily relied on prompting-based methods (Wang et al., 2024b; Sodhi et al., 2024; Fu et al., 2024; Zhang et al., 2025; Yang et al., 2025b) or behavior cloning (BC), which imi-tates demonstrated trajectories via supervised fine-tuning (Yin et al., 2024; Hong et al.,…”
Large Language Models Can Self-Improve At Web Agent Tasks
Patel, A., Hofmarcher, M., Leoveanu-Condrei, C., Dinu, M.-C., Callison-Burch, C., & Hochreiter, S. (2024) · arXiv · doi:10.48550/arxiv.2405.20309 · Cited by 1 · cited 1×
“…interactive tasks, including web navigation (Nakano et al., 2021; Yao et al., 2022; Ma et al., 2023; Gur et al., 2024; Abuelsaad et al., 2024; Lutz et al., 2024; Pa-tel et al., 2024; Putta et al., 2024), general computer use (Li et al., 2020; Deng et al., 2023; Yang et al., 2024), and…”
Research works printed but not linked: 18
- Zhendong Chu, Shen Wang, Jian Xie, Tinghui Zhu, Yibo Yan, Jinheng Ye, Aoxiao Zhong, Xuming Hu, Jing Liang, Philip S Yu, and 1 others. 2025. LLM agents for education: Advances and applications. arXiv preprint arXiv:2503.11733. · arXiv
- Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shi- rong Ma, Peiyi Wang, Xiao Bi, and 1 others. 2025. Deepseek-R1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948. · arXiv
- Hongliang He, Wenlin Yao, Kaixin Ma, Wenhao Yu, Yong Dai, Hongming Zhang, Zhenzhong Lan, and Dong Yu. 2024a. Webvoyager: Building an end-to- end web agent with large multimodal models. In Proceedings of the 62nd Annual Meeting of the As- sociation for Computational Linguistics (Volume 1: Long Papers), pages 6864-6890.
- Wenyi Hong, Weihan Wang, Qingsong Lv, Jiazheng Xu, Wenmeng Yu, Junhui Ji, Yan Wang, Zihan Wang, Yuxiao Dong, Ming Ding, and 1 others. 2024. Coga- gent: A visual language model for gui agents. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14281-14290.
- PMLR.
- Kenton Lee, Mandar Joshi, Iulia Raluca Turc, Hexi- ang Hu, Fangyu Liu, Julian Martin Eisenschlos, Ur- vashi Khandelwal, Peter Shaw, Ming-Wei Chang, and Kristina Toutanova. 2023. Pix2struct: Screen- shot parsing as pretraining for visual language under- standing. In International Conference on Machine Learning, pages 18893-18912.
- PMLR.
- Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, and 1 others. 2021. WebGPT: Browser-assisted question-answering with human feedback. arXiv preprint arXiv:2112.09332. · arXiv
- Liangbo Ning, Ziran Liang, Zhuohang Jiang, Haohao Qu, Yujuan Ding, Wenqi Fan, Xiao-yong Wei, Shanru Lin, Hui Liu, Philip S Yu, and 1 others. 2025. A sur- vey of webagents: Towards next-generation ai agents for web automation with large foundation models. arXiv preprint arXiv:2503.23350. · arXiv
- Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. 2020. Zero: Memory optimizations toward training trillion parameter models. In SC20: International Conference for High Performance Com- puting, Networking, Storage and Analysis, pages 1- 16. IEEE.
- Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, and 1 others. 2024. Deepseek- math: Pushing the limits of mathematical reason- ing in open language models. arXiv preprint arXiv:2402.03300. · arXiv
- Peter Shaw, Mandar Joshi, James Cohan, Jonathan Be- rant, Panupong Pasupat, Hexiang Hu, Urvashi Khan- delwal, Kenton Lee, and Kristina N Toutanova. 2023. From pixels to ui actions: Learning to follow in- structions via graphical user interfaces. Advances in Neural Information Processing Systems, 36:34354- 34370.
- Mohit Shridhar, Jesse Thomason, Daniel Gordon, Yonatan Bisk, Winson Han, Roozbeh Mottaghi, Luke Zettlemoyer, and Dieter Fox. 2020. Alfred: A bench- mark for interpreting grounded instructions for ev- eryday tasks. In Proceedings of the IEEE/CVF con- ference on computer vision and pattern recognition, pages 10740-10749.
- Huatong Song, Jinhao Jiang, Yingqian Min, Jie Chen, Zhipeng Chen, Wayne Xin Zhao, Lei Fang, and Ji- Rong Wen. 2025. R1-searcher: Incentivizing the search capability in LLMs via reinforcement learning. arXiv preprint arXiv:2503.05592. · arXiv
- Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, and 1 others. 2025. Kimi k1. 5: Scaling reinforcement learning with llms. arXiv preprint arXiv:2501.12599. · arXiv
- Zora Zhiruo Wang, Jiayuan Mao, Daniel Fried, and Graham Neubig. 2024b. Agent workflow memory. arXiv preprint arXiv:2409.07429. · arXiv
- Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Beibin Li, Erkang Zhu, Li Jiang, Xiaoyun Zhang, Shaokun Zhang, Jiale Liu, Ahmed Hassan Awadallah, Ryen W White, Doug Burger, and Chi Wang. 2024. Autogen: Enabling next-gen LLM applications via multi-agent conversations. In First Conference on Language Modeling.
- Ke Yang, Yao Liu, Sapana Chaudhary, Rasool Fakoor, Pratik Chaudhari, George Karypis, and Huzefa Rang- wala. 2025b. AgentOccam: A simple yet strong base- line for LLM-based web agents. In The Thirteenth International Conference on Learning Representa- tions.
Cited by
Sorted by citations
RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement LearningPreprint
Wang, Z., Wang, K., Wang, Q., Zhang, P., Li, L., Yang, Z., Jin, X., Yu, K. F., Nguyen, M. N., Liu, L., et al. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2504.20073 · Cited by 443
“Stale or misaligned rollouts can induce collapse, while maintaining freshness—alongside limits on action granularity and sufficient task diversity—enables stable and effective RL training.”
Self-citationLarge Language Models for Information Retrieval: A Survey
Zhu, Y., Yuan, H., Wang, S., Liu, J., Liu, W., Deng, C., Chen, H., Liu, Z., Dou, Z., & Wen, J.-R. (2026) · ACM Transactions on Information Systems · doi:10.1145/3748304 · Cited by 225
“Traditional evaluation metrics for ranking primarily focus on comparing the retrieval results of IR models with ground-truth (relevance) labels.”
WebDancer: Towards Autonomous Information Seeking Agency
Wu, J.-L., Li, B., Fang, R.-N., et al. (2025) · Neural Information Processing Systems · doi:10.48550/arxiv.2505.22648 · Cited by 204
“The SFT stage offers strong initialization for the subsequent RL stage [29].”
The Surprising Effectiveness of Negative Reinforcement in LLM Reasoning
Zhu, X., Xia, M., Wei, Z., et al. (2025) · Neural Information Processing Systems · doi:10.48550/arxiv.2506.01347 · Cited by 181
“…scaling has emerged as a promising direction for enhancing model performance [2, 3, 4, 29, 35, 38, 42, 46, 49, 50, 51, 56, 59, 80, 67, 73, 78, 55]— particularly in reasoning tasks where generating multiple candidate solutions [7, 80] or longer reasoning traces [35, 68, 75] can help hit the…”
Self-citationOne Token to Fool LLM-as-a-Judge
Zhao, Y., Liu, H., Yu, D., Kung, S., Chen, M., Mi, H., & Yu, D. (2025) · arXiv · doi:10.48550/arxiv.2507.08794 · Cited by 77
Metadata
- Authors
- Wei, Z., Yao, W., Liu, Y., Zhang, W., Lu, Q., Qiu, L., Yu, C., Xu, P., Zhang, C., Yin, B., Yun, H., & Li, L.
- Affiliations
- Georgia Institute of Technology (US)
- University of Virginia (US)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2025
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2505.16421
- OpenAlex
- W4416451824
- Semantic Scholar
- a343a24a0d47e0acf4d71bfd0215924bc76d0256
- Cited by
- 0
- References
- 64
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Information Systems › Web Data Mining and Analysis
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
- Keywords
- web agents, multi-turn reinforcement learning, large language models, end-to-end learning, long-horizon decision-making, chain-of-thought reasoning, behavior cloning, warm-up, test-time scaling
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
