WebAgent-R1: Training Web Agents via End-to-End Multi-Turn Reinforcement Learning
Open accessAbstract
While reinforcement learning (RL) has demonstrated remarkable success in enhancing large language models (LLMs), it has primarily focused on single-turn tasks such as solving math problems. Training effective web agents for multi-turn interactions remains challenging due to the complexity of long-horizon decision-making across dynamic web interfaces. In this work, we present WebAgent-R1, a simple yet effective end-to-end multi-turn RL framework for training web agents. It learns directly from online interactions with web environments by asynchronously generating diverse trajectories, entirely guided by binary rewards depending on task success. Experiments on the WebArena-Lite benchmark demonstrate the effectiveness of WebAgent-R1, boosting the task success rate of Qwen-2.5-3B from 6.1% to 33.9% and Llama-3.1-8B from 8.5% to 44.8%, significantly outperforming existing state-of-the-art methods and strong proprietary models such as OpenAI o3. In-depth analyses reveal the effectiveness of the thinking-based prompting strategy and test-time scaling through increased interactions for web tasks. We further investigate different RL initialization policies by introducing two variants, namely WebAgent-R1-Zero and WebAgent-R1-CoT, which highlight the importance of the warm-up training stage (i.e., behavior cloning) and provide insights on incorporating long chain-of-thought (CoT) reasoning in web agents.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Reinforcement learning has recently shown strong potential for aligning LLM behavior with high-level objectives, including tool use, web navigation, and search optimization.
“Reinforcement learning (RL), in contrast, has recently shown strong potential for aligning LLM behavior with high-level objectives, including tool use (Qian et al., 2025; Wang et al., 2025), web navigation (Wei et al., 2025), and search optimization (Jin et al., 2025; Song et al., 2025).”
From Memory-R1: Enhancing Large Language Model Agents to Manage and Utilize Memories via Reinforcement Learning (2025) · cites this paper as Wei et al., 2025
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Proximal Policy Optimization Algorithms
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., & Klimov, O. (2017) · arXiv · doi:10.48550/arxiv.1707.06347 · Cited by 10,625 · cited 1× · see citing works
“This ensures that the learning process is better aligned with the agent’s most recent actions, often leading to more stable and effective learning (Schulman et al., 2015, 2017).”
High-Dimensional Continuous Control Using Generalized Advantage Estimation
Schulman, J., Moritz, P., Levine, S., Jordan, M., & Abbeel, P. (2015) · arXiv · doi:10.48550/arxiv.1506.02438 · Cited by 2,103 · cited 1× · see citing works
“This ensures that the learning process is better aligned with the agent’s most recent actions, often leading to more stable and effective learning (Schulman et al., 2015, 2017).”
A survey on large language model based autonomous agents
Wang, L., Ma, C., Feng, X., Zhang, Z., Yang, H., Zhang, J., Chen, Z., Tang, J., Chen, X., Lin, Y., et al. (2024) · Frontiers of Computer Science · doi:10.1007/s11704-024-40231-1 · Cited by 1,941 · cited 1× · see citing works
“For a comprehensive overview, we refer readers to recent surveys (Wang et al., 2024a; Tseng et al., 2024; Hu et al., 2025; Ning et al., 2025).”
Self-citationEfficient Memory Management for Large Language Model Serving with PagedAttention
Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez, J., Zhang, H., & Stoica, I. (2023) · Proceedings of the 29th Symposium on Operating Systems Principles · doi:10.1145/3600006.3613165 · Cited by 1,817 · cited 1× · see citing works
“(Kwon et al., 2023) with a tensor parallel size of 1 and a GPU memory utilization ratio of 0.7.”
InfluentialSelf-citation(2020) · doi:10.1109/sc41405.2020.00024 · Cited by 928 · cited 1× · see citing works
“To optimize GPU utilization, we adopt DeepSpeed (Rajbhandari et al., 2020) for distributed training with ZeRO-3 offload, along with gradient checkpointing, FlashAttention-2 (Dao, 2024), and bf16 mixed precision training enabled for computation efficiency.”
ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Tasks
Shridhar, M., Thomason, J., Gordon, D., Bisk, Y., Han, W., Mottaghi, R., Zettlemoyer, L., & Fox, D. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.01075 · Cited by 540 · cited 1× · see citing works
“…2024; Abuelsaad et al., 2024; Lutz et al., 2024; Pa-tel et al., 2024; Putta et al., 2024), general computer use (Li et al., 2020; Deng et al., 2023; Yang et al., 2024), and embodied environments (Puig et al., 2018; Shridhar et al., 2020; Toyama et al., 2021; Fan et al., 2022; Huang et al., 2022).”
VirtualHome: Simulating Household Activities Via Programs
Puig, X., Ra, K., Boben, M., Li, J., Wang, T., Fidler, S., & Torralba, A. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00886 · Cited by 378 · cited 1× · see citing works
“…2024; Abuelsaad et al., 2024; Lutz et al., 2024; Pa-tel et al., 2024; Putta et al., 2024), general computer use (Li et al., 2020; Deng et al., 2023; Yang et al., 2024), and embodied environments (Puig et al., 2018; Shridhar et al., 2020; Toyama et al., 2021; Fan et al., 2022; Huang et al., 2022).”
Camel: Communicative agents for" mind" exploration of large language model society
Li, G., Hammoud, H., Itani, H., Khizbullin, D., & Ghanem, B. (2023) · Advances in Neural Information Processing Systems 36 · doi:10.52202/075280-2264 · Cited by 314 · cited 1× · see citing works
“LLMs have demonstrated promising agentic capabilities, such as breaking down complex tasks into manageable subgoals and reasoning over long horizons (Zhou et al., 2022; Huang et al., 2022; Madaan et al., 2022; Li et al., 2023a,b; Wu et al., 2024; Liu et al., 2025b; Chu et al., 2025).”
Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agents
Huang, W., Abbeel, P., Pathak, D., & Mordatch, I. (2022) · arXiv · doi:10.48550/arxiv.2201.07207 · Cited by 304 · cited 2× · see citing works
“…2024; Abuelsaad et al., 2024; Lutz et al., 2024; Pa-tel et al., 2024; Putta et al., 2024), general computer use (Li et al., 2020; Deng et al., 2023; Yang et al., 2024), and embodied environments (Puig et al., 2018; Shridhar et al., 2020; Toyama et al., 2021; Fan et al., 2022; Huang et al., 2022).”
Language Models of Code are Few-Shot Commonsense Learners
Madaan, A., Zhou, S., Alon, U., et al. (2022) · Conference on Empirical Methods in Natural Language Processing · doi:10.48550/arxiv.2210.07128 · Cited by 285 · cited 1×
“LLMs have demonstrated promising agentic capabilities, such as breaking down complex tasks into manageable subgoals and reasoning over long horizons (Zhou et al., 2022; Huang et al., 2022; Madaan et al., 2022; Li et al., 2023a,b; Wu et al., 2024; Liu et al., 2025b; Chu et al., 2025).”
ZeroSearch: Incentivize the Search Capability of LLMs without Searching
Sun, H., Qiao, Z.-L., Guo, J., et al. (2025) · arXiv.org · doi:10.48550/arxiv.2505.04588 · Cited by 222 · cited 1×
“Recent efforts have made some progress in this direction, such as training LLM agents to repeatedly use search engines (Jin et al., 2025; Sun et al., 2025; Chen et al., 2025; Song et al., 2025; Wang et al., 2025a), but typically constrain actions to simple API calls without real environment…”
The Surprising Effectiveness of Negative Reinforcement in LLM Reasoning
Zhu, X., Xia, M., Wei, Z., et al. (2025) · Neural Information Processing Systems · doi:10.48550/arxiv.2506.01347 · Cited by 181
Self-citationAdvantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement LearningPreprint
Peng, X. B., Kumar, A., Zhang, G., & Levine, S. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1910.00177 · Cited by 158 · cited 2× · see citing works
“However, most of this line of research has heavily relied on offline or iterative off-policy RL solutions (Peng et al., 2019; Pan et al., 2024; Qi et al., 2025), which break the end-to-end interaction between the web agent and environment, and introduce additional complexities such as trajectory…”
FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning
Dao, T. (2023) · arXiv · doi:10.48550/arxiv.2307.08691 · Cited by 143 · see citing works
Web Agents with World Models: Learning and Leveraging Environment Dynamics in Web NavigationPreprint
Chae, H., Kim, N., Ong, K. T.-I., et al. (2024) · International Conference on Learning Representations · doi:10.48550/arxiv.2410.13232 · Cited by 143 · cited 1× · see citing works
“Their effectiveness in multi-turn, interactive environments—particularly in complex scenarios requiring long-horizon decision-making and domain-specific skills, such as web browsing (Zhou et al., 2024a; He et al., 2024a; Chae et al., 2025)— still remains underexplored.”
Mind2Web: Towards a Generalist Agent for the Web
Deng, X., Gu, Y., Zheng, B., Chen, S., Stevens, S., Wang, B., Sun, H., & Su, Y. (2023) · arXiv · doi:10.48550/arxiv.2306.06070 · Cited by 134 · cited 2× · see citing works
“Following prior work (Qi et al., 2025), we adopt WebArena (Zhou et al., 2024a) as the web environment over other simulated or static environments such as WebShop (Yao et al., 2022) or Mind2Web (Deng et al., 2023) for greater practicality—”
InfluentialZhang, Y., Ma, Z.-J., Ma, Y., et al. (2024) · AAAI Conference on Artificial Intelligence · doi:10.48550/arxiv.2408.15978 · Cited by 127 · cited 1×
“Early works on web agents primarily relied on prompting-based methods (Wang et al., 2024b; Sodhi et al., 2024; Fu et al., 2024; Zhang et al., 2025; Yang et al., 2025b) or behavior cloning (BC), which imi-tates demonstrated trajectories via supervised fine-tuning (Yin et al., 2024; Hong et al.,…”
SWEET-RL: Training Multi-Turn LLM Agents on Collaborative Reasoning TasksPreprint
Zhou, Y., Jiang, S., Tian, Y.-D., et al. (2025) · arXiv.org · doi:10.48550/arxiv.2503.15478 · Cited by 114 · see citing works
WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agents
Yao, S., Chen, H., Yang, J., & Narasimhan, K. (2022) · arXiv · doi:10.48550/arxiv.2207.01206 · Cited by 107 · cited 2× · see citing works
“Following prior work (Qi et al., 2025), we adopt WebArena (Zhou et al., 2024a) as the web environment over other simulated or static environments such as WebShop (Yao et al., 2022) or Mind2Web (Deng et al., 2023) for greater practicality—”
InfluentialFu, Y., Kim, D.-K., Kim, J., Sohn, S., Logeswaran, L., Bae, K., & Lee, H. (2024) · Neural Information Processing Systems · doi:10.48550/arxiv.2403.08978 · Cited by 98 · cited 1×
“Early works on web agents primarily relied on prompting-based methods (Wang et al., 2024b; Sodhi et al., 2024; Fu et al., 2024; Zhang et al., 2025; Yang et al., 2025b) or behavior cloning (BC), which imi-tates demonstrated trajectories via supervised fine-tuning (Yin et al., 2024; Hong et al.,…”
Two Tales of Persona in LLMs: A Survey of Role-Playing and Personalization
Tseng, Y.-M., Huang, Y.-C., Hsiao, T.-Y., Chen, W.-L., Huang, C.-W., Meng, Y., & Chen, Y.-N. (2024) · arXiv · doi:10.48550/arxiv.2406.01171 · Cited by 87
Scaffolding Coordinates to Promote Vision-Language Coordination in Large Multi-Modal Models
Lei, X.-Y., Yang, Z.-H., Chen, X.-R., et al. (2024) · International Conference on Computational Linguistics · doi:10.48550/arxiv.2402.12058 · Cited by 81 · cited 1×
“…this line of work, GUI agents leverage additional multimodal inputs such as screenshots, enabling visual-guided interactions with the environment (Lee et al., 2023; Shaw et al., 2023; Zheng et al., 2024; He et al., 2024a,b; Koh et al., 2024; Kil et al., 2024; Lei et al., 2025; Liu et al., 2025a).”
Self-citationMapping Natural Language Instructions to Mobile UI Action Sequences
Li, Y., He, J., Zhou, X., Zhang, Y., & Baldridge, J. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.729 · Cited by 77 · cited 1× · see citing works
“…2022; Ma et al., 2023; Gur et al., 2024; Abuelsaad et al., 2024; Lutz et al., 2024; Pa-tel et al., 2024; Putta et al., 2024), general computer use (Li et al., 2020; Deng et al., 2023; Yang et al., 2024), and embodied environments (Puig et al., 2018; Shridhar et al., 2020; Toyama et al., 2021; Fan…”
SWE-bench: Can Language Models Resolve Real-World GitHub Issues?
Jimenez, C. E., Yang, J., Wettig, A., Yao, S., Pei, K., Press, O., & Narasimhan, K. (2023) · arXiv · doi:10.48550/arxiv.2310.06770 · Cited by 72 · cited 2× · see citing works
“…2024; Abuelsaad et al., 2024; Lutz et al., 2024; Pa-tel et al., 2024; Putta et al., 2024), general computer use (Li et al., 2020; Deng et al., 2023; Yang et al., 2024), and embodied environments (Puig et al., 2018; Shridhar et al., 2020; Toyama et al., 2021; Fan et al., 2022; Huang et al., 2022).”
MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledge
Fan, L., Wang, G., Jiang, Y., Mandlekar, A., Yang, Y., Zhu, H., Tang, A., Huang, D.-A., Zhu, Y., & Anandkumar, A. (2022) · arXiv · doi:10.48550/arxiv.2206.08853 · Cited by 72 · cited 1× · see citing works
“…2024; Abuelsaad et al., 2024; Lutz et al., 2024; Pa-tel et al., 2024; Putta et al., 2024), general computer use (Li et al., 2020; Deng et al., 2023; Yang et al., 2024), and embodied environments (Puig et al., 2018; Shridhar et al., 2020; Toyama et al., 2021; Fan et al., 2022; Huang et al., 2022).”
LASER: LLM Agent with State-Space Exploration for Web Navigation
Ma, K., Zhang, H., Wang, H., Pan, X., & Yu, D. (2023) · arXiv.org · doi:10.48550/arxiv.2309.08172 · Cited by 72 · cited 1×
“…LLM-based agents have been applied to a variety of real-world interactive tasks, including web navigation (Nakano et al., 2021; Yao et al., 2022; Ma et al., 2023; Gur et al., 2024; Abuelsaad et al., 2024; Lutz et al., 2024; Pa-tel et al., 2024; Putta et al., 2024), general computer use (Li et…”
AutoWebGLM: A Large Language Model-based Web Navigating Agent
Lai, H., Liu, X., Iong, I. L., Yao, S., Chen, Y., Shen, P., Yu, H., Zhang, H., Zhang, X., Dong, Y., et al. (2024) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3637528.3671620 · Cited by 69 · cited 1× · see citing works
“…methods (Wang et al., 2024b; Sodhi et al., 2024; Fu et al., 2024; Zhang et al., 2025; Yang et al., 2025b) or behavior cloning (BC), which imi-tates demonstrated trajectories via supervised fine-tuning (Yin et al., 2024; Hong et al., 2024; Lai et al., 2024; He et al., 2024b; Putta et al., 2024).”
OS Agents: A Survey on MLLM-based Agents for Computer, Phone and Browser Use
Hu, X., Xiong, T., Yi, B., et al. (2025) · Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2025.acl-long.369 · Cited by 57 · cited 2×
“For a comprehensive overview, we refer readers to recent surveys (Wang et al., 2024a; Tseng et al., 2024; Hu et al., 2025; Ning et al., 2025).”
Self-citationHe, H., Yao, W., Ma, K.-X., et al. (2024) · Annual Meeting of the Association for Computational Linguistics · doi:10.48550/arxiv.2410.19609 · Cited by 50 · cited 1×
“…methods (Wang et al., 2024b; Sodhi et al., 2024; Fu et al., 2024; Zhang et al., 2025; Yang et al., 2025b) or behavior cloning (BC), which imi-tates demonstrated trajectories via supervised fine-tuning (Yin et al., 2024; Hong et al., 2024; Lai et al., 2024; He et al., 2024b; Putta et al., 2024).”
Self-citationPix2Struct: Screenshot Parsing as Pretraining for Visual Language UnderstandingPreprint
Lee, K., Joshi, M., Turc, I., et al. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2210.03347 · Cited by 47 · cited 1× · see citing works
“Complementary to this line of work, GUI agents leverage additional multimodal inputs such as screenshots, enabling visual-guided interactions with the environment (Lee et al., 2023; Shaw et al., 2023; Zheng et al., 2024; He et al., 2024a,b; Koh et al., 2024; Kil et al., 2024; Lei et al., 2025; Liu…”
Show Me More Details: Discovering Hierarchies of Procedures from Semi-structured Web Data
Zhou, S., Zhang, L., Yang, Y., et al. (2022) · Annual Meeting of the Association for Computational Linguistics · doi:10.48550/arxiv.2203.07264 · Cited by 33 · cited 1×
“LLMs have demonstrated promising agentic capabilities, such as breaking down complex tasks into manageable subgoals and reasoning over long horizons (Zhou et al., 2022; Huang et al., 2022; Madaan et al., 2022; Li et al., 2023a,b; Wu et al., 2024; Liu et al., 2025b; Chu et al., 2025).”
WebArena: A Realistic Web Environment for Building Autonomous Agents
Zhou, S., Xu, F. F., Zhu, H., Zhou, X., Lo, R., Sridhar, A., Cheng, X., Ou, T., Bisk, Y., Fried, D., et al. (2023) · arXiv · doi:10.48550/arxiv.2307.13854 · Cited by 31 · cited 16× · see citing works
“Empirically, W EB A GENT -R1 achieves new state-of-the-art re-sults on the WebArena-Lite benchmark.”
InfluentialSWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering
Yang, J., Jimenez, C. E., Wettig, A., Lieret, K., Yao, S., Narasimhan, K., & Press, O. (2024) · arXiv · doi:10.48550/arxiv.2405.15793 · Cited by 31 · see citing works
Agent-E: From Autonomous Web Navigation to Foundational Design Principles in Agentic Systems
Abuelsaad, T., Akkil, D., Dey, P., Jagmohan, A., Vempaty, A., & Kokku, R. (2024) · arXiv · doi:10.48550/arxiv.2407.13032 · Cited by 31 · cited 1× · see citing works
“…to a variety of real-world interactive tasks, including web navigation (Nakano et al., 2021; Yao et al., 2022; Ma et al., 2023; Gur et al., 2024; Abuelsaad et al., 2024; Lutz et al., 2024; Pa-tel et al., 2024; Putta et al., 2024), general computer use (Li et al., 2020; Deng et al., 2023; Yang…”
SelfElicit: Your Language Model Secretly Knows Where is the Relevant Evidence
Liu, Z., Amjad, R. A., Adkathimar, R., et al. (2025) · Annual Meeting of the Association for Computational Linguistics · doi:10.48550/arxiv.2502.08767 · Cited by 30 · cited 1×
“LLMs have demonstrated promising agentic capabilities, such as breaking down complex tasks into manageable subgoals and reasoning over long horizons (Zhou et al., 2022; Huang et al., 2022; Madaan et al., 2022; Li et al., 2023a,b; Wu et al., 2024; Liu et al., 2025b; Chu et al., 2025).”
WILBUR: Adaptive In-Context Learning for Robust and Accurate Web Agents
Lutz, M., Bohra, A., Saroyan, M., et al. (2024) · arXiv.org · doi:10.48550/arxiv.2404.05902 · Cited by 24 · cited 1×
“…interactive tasks, including web navigation (Nakano et al., 2021; Yao et al., 2022; Ma et al., 2023; Gur et al., 2024; Abuelsaad et al., 2024; Lutz et al., 2024; Pa-tel et al., 2024; Putta et al., 2024), general computer use (Li et al., 2020; Deng et al., 2023; Yang et al., 2024), and…”
DigiRL: Training In-The-Wild Device-Control Agents with Autonomous Reinforcement Learning
Bai, H., Zhou, Y., Cemri, M., Pan, J., Suhr, A., Levine, S., & Kumar, A. (2024) · arXiv · doi:10.48550/arxiv.2406.11896 · Cited by 20 · cited 3×
“…2024; Qi et al., 2025), which break the end-to-end interaction between the web agent and environment, and introduce additional complexities such as trajectory filtering (Bai et al., 2024), outcome reward model training (Qi et al., 2025), or iterative optimization procedures (Zhou et al., 2024b).”
ArCHer: Training Language Model Agents via Hierarchical Multi-Turn RLPreprint
Zhou, Y., Zanette, A., Pan, J., Levine, S., & Kumar, A. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2402.19446 · Cited by 18 · cited 2× · see citing works
“…2024; Qi et al., 2025), which break the end-to-end interaction between the web agent and environment, and introduce additional complexities such as trajectory filtering (Bai et al., 2024), outcome reward model training (Qi et al., 2025), or iterative optimization procedures (Zhou et al., 2024b).”
A Real-World WebAgent with Planning, Long Context Understanding, and Program Synthesis
Gur, I., Furuta, H., Huang, A., Safdari, M., Matsuo, Y., Eck, D., & Faust, A. (2023) · arXiv · doi:10.48550/arxiv.2307.12856 · Cited by 17 · cited 1× · see citing works
“…have been applied to a variety of real-world interactive tasks, including web navigation (Nakano et al., 2021; Yao et al., 2022; Ma et al., 2023; Gur et al., 2024; Abuelsaad et al., 2024; Lutz et al., 2024; Pa-tel et al., 2024; Putta et al., 2024), general computer use (Li et al., 2020; Deng et…”
Agent Lumos: Unified and Modular Training for Open-Source Language Agents
Yin, D., Brahman, F., Ravichander, A., Chandu, K., Chang, K.-W., Choi, Y., & Lin, B. Y. (2024) · Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2024.acl-long.670 · Cited by 17 · cited 1× · see citing works
“…methods (Wang et al., 2024b; Sodhi et al., 2024; Fu et al., 2024; Zhang et al., 2025; Yang et al., 2025b) or behavior cloning (BC), which imi-tates demonstrated trajectories via supervised fine-tuning (Yin et al., 2024; Hong et al., 2024; Lai et al., 2024; He et al., 2024b; Putta et al., 2024).”
Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning
Jin, B., Zeng, H., Yue, Z., Yoon, J., Arik, S., Wang, D., Zamani, H., & Han, J. (2025) · arXiv · doi:10.48550/arxiv.2503.09516 · Cited by 13 · cited 1× · see citing works
“Recent efforts have made some progress in this direction, such as training LLM agents to repeatedly use search engines (Jin et al., 2025; Sun et al., 2025; Chen et al., 2025; Song et al., 2025; Wang et al., 2025a), but typically constrain actions to simple API calls without real environment…”
GPT-4V(ision) is a Generalist Web Agent, if Grounded
Zheng, B., Gou, B., Kil, J., Sun, H., & Su, Y. (2024) · arXiv · doi:10.48550/arxiv.2401.01614 · Cited by 10 · cited 1× · see citing works
“…this line of work, GUI agents leverage additional multimodal inputs such as screenshots, enabling visual-guided interactions with the environment (Lee et al., 2023; Shaw et al., 2023; Zheng et al., 2024; He et al., 2024a,b; Koh et al., 2024; Kil et al., 2024; Lei et al., 2025; Liu et al., 2025a).”
Agent Q: Advanced Reasoning and Learning for Autonomous AI Agents
Putta, P., Mills, E., Garg, N., Motwani, S., Finn, C., Garg, D., & Rafailov, R. (2024) · arXiv · doi:10.48550/arxiv.2408.07199 · Cited by 10 · cited 2× · see citing works
“…(Nakano et al., 2021; Yao et al., 2022; Ma et al., 2023; Gur et al., 2024; Abuelsaad et al., 2024; Lutz et al., 2024; Pa-tel et al., 2024; Putta et al., 2024), general computer use (Li et al., 2020; Deng et al., 2023; Yang et al., 2024), and embodied environments (Puig et al., 2018;…”
RAST: Reasoning Activation in LLMs via Small-model Transfer
Ouyang, S., Zhu, X.-Y., Xiao, Z.-L., et al. (2025) · Neural Information Processing Systems · doi:10.48550/arxiv.2506.15710 · Cited by 10 · cited 1×
“However, most prior work focuses on single-turn tasks such as math problems (Shao et al., 2024; Zhu et al., 2025; Shao et al., 2025; Ouyang et al., 2025; Wei et al., 2025), with lim-8 ited exploration in multi-turn settings (Zhou et al., 2024b, 2025).”
AndroidEnv: A Reinforcement Learning Platform for AndroidPreprint
Toyama, D., Hamel, P., Gergely, A., Comanici, G., Glaese, A., Ahmed, Z., Jackson, T., Mourad, S., & Precup, D. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2105.13231 · Cited by 8 · cited 1× · see citing works
“…2024; Abuelsaad et al., 2024; Lutz et al., 2024; Pa-tel et al., 2024; Putta et al., 2024), general computer use (Li et al., 2020; Deng et al., 2023; Yang et al., 2024), and embodied environments (Puig et al., 2018; Shridhar et al., 2020; Toyama et al., 2021; Fan et al., 2022; Huang et al., 2022).”
Take a Break in the Middle: Investigating Subgoals towards Hierarchical Script Generation
Li, X., Cao, Y., Chen, M., et al. (2023) · Annual Meeting of the Association for Computational Linguistics · doi:10.48550/arxiv.2305.10907 · Cited by 8
Beyond Outcome Reward: Decoupling Search and Answering Improves LLM Agents
Wang, Y., Wei, Z., Zhu, X.-Y., et al. (2025) · arXiv.org · doi:10.48550/arxiv.2510.04695 · Cited by 6 · cited 1×
“…efforts have made some progress in this direction, such as training LLM agents to repeatedly use search engines (Jin et al., 2025; Sun et al., 2025; Chen et al., 2025; Song et al., 2025; Wang et al., 2025a), but typically constrain actions to simple API calls without real environment interaction.”
Self-citationDual-View Visual Contextualization for Web Navigation
Kil, J., Song, C. H., Zheng, B., Deng, X., Su, Y., & Chao, W. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01369 · Cited by 5 · cited 1× · see citing works
“…this line of work, GUI agents leverage additional multimodal inputs such as screenshots, enabling visual-guided interactions with the environment (Lee et al., 2023; Shaw et al., 2023; Zheng et al., 2024; He et al., 2024a,b; Koh et al., 2024; Kil et al., 2024; Lei et al., 2025; Liu et al., 2025a).”
RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement LearningPreprint
Wang, Z., Wang, K., Wang, Q., Zhang, P., Li, L., Yang, Z., Jin, X., Yu, K. F., Nguyen, M. N., Liu, L., et al. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2504.20073 · Cited by 3 · see citing works
Self-citationAutonomous Evaluation and Refinement of Digital Agents
Pan, J., Zhang, Y., Tomlin, N., Zhou, Y., Levine, S., & Suhr, A. (2024) · arXiv · doi:10.48550/arxiv.2404.06474 · Cited by 3 · cited 1× · see citing works
“However, most of this line of research has heavily relied on offline or iterative off-policy RL solutions (Peng et al., 2019; Pan et al., 2024; Qi et al., 2025), which break the end-to-end interaction between the web agent and environment, and introduce additional complexities such as trajectory…”
SimpleRL-Zoo: Investigating and Taming Zero Reinforcement Learning for Open Base Models in the WildPreprint
Zeng, W., Huang, Y., Liu, Q., Liu, W., He, K., Ma, Z., & He, J. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2503.18892 · Cited by 2 · cited 1× · see citing works
“However, existing works have primarily focused on single-turn, non-interactive tasks such as mathematical reasoning (Shao et al., 2024; Zeng et al., 2025).”
VisualWebArena: Evaluating Multimodal Agents on Realistic Visual Web Tasks
Koh, J. Y., Lo, R., Jang, L., Duvvur, V., Lim, M. C., Huang, P.-Y., Neubig, G., Zhou, S., Salakhutdinov, R., & Fried, D. (2024) · arXiv · doi:10.48550/arxiv.2401.13649 · Cited by 1 · cited 1× · see citing works
“…this line of work, GUI agents leverage additional multimodal inputs such as screenshots, enabling visual-guided interactions with the environment (Lee et al., 2023; Shaw et al., 2023; Zheng et al., 2024; He et al., 2024a,b; Koh et al., 2024; Kil et al., 2024; Lei et al., 2025; Liu et al., 2025a).”
Spurious Rewards: Rethinking Training Signals in RLVR
Shao, R., Li, S. S., Xin, R., Geng, S., Wang, Y., Oh, S., Du, S. S., Lambert, N., Min, S., Krishna, R., et al. (2025) · arXiv · doi:10.48550/arxiv.2506.10947 · Cited by 1 · cited 1×
“However, most prior work focuses on single-turn tasks such as math problems (Shao et al., 2024; Zhu et al., 2025; Shao et al., 2025; Ouyang et al., 2025; Wei et al., 2025), with lim-8 ited exploration in multi-turn settings (Zhou et al., 2024b, 2025).”
VisualAgentBench: Towards Large Multimodal Models as Visual Foundation AgentsPreprint
Liu, X., Zhang, T., Gu, Y., Iong, I. L., Xu, Y., Song, X., Zhang, S., Lai, H., Liu, X., Zhao, H., et al. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2408.06327 · Cited by 1 · cited 12× · see citing works
“3 Experiments”
InfluentialSteP: Stacked LLM Policies for Web Actions
Sodhi, P., Branavan, S. R. K., Artzi, Y., & McDonald, R. (2023) · arXiv · doi:10.48550/arxiv.2310.03720 · Cited by 1 · cited 1× · see citing works
“Early works on web agents primarily relied on prompting-based methods (Wang et al., 2024b; Sodhi et al., 2024; Fu et al., 2024; Zhang et al., 2025; Yang et al., 2025b) or behavior cloning (BC), which imi-tates demonstrated trajectories via supervised fine-tuning (Yin et al., 2024; Hong et al.,…”
Large Language Models Can Self-Improve At Web Agent Tasks
Patel, A., Hofmarcher, M., Leoveanu-Condrei, C., Dinu, M.-C., Callison-Burch, C., & Hochreiter, S. (2024) · arXiv · doi:10.48550/arxiv.2405.20309 · Cited by 1 · cited 1× · see citing works
“…interactive tasks, including web navigation (Nakano et al., 2021; Yao et al., 2022; Ma et al., 2023; Gur et al., 2024; Abuelsaad et al., 2024; Lutz et al., 2024; Pa-tel et al., 2024; Putta et al., 2024), general computer use (Li et al., 2020; Deng et al., 2023; Yang et al., 2024), and…”
WebRL: Training LLM Web Agents via Self-Evolving Online Curriculum Reinforcement LearningPreprint
Qi, Z., Liu, X., Iong, I. L., Lai, H., Sun, X., Zhao, W., Yu, Y., Yang, X., Sun, J., Yao, S., et al. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2411.02337 · cited 18×
“However, most of this line of research has heavily relied on offline or iterative off-policy RL solutions (Peng et al., 2019; Pan et al., 2024; Qi et al., 2025), which break the end-to-end interaction between the web agent and environment, and introduce additional complexities such as trajectory…”
Influential
Flagged first, then by citations
7 of the 64 references couldn’t be matched to a known paper.
Research works printed but not linked: 18
- Zhendong Chu, Shen Wang, Jian Xie, Tinghui Zhu, Yibo Yan, Jinheng Ye, Aoxiao Zhong, Xuming Hu, Jing Liang, Philip S Yu, and 1 others. 2025. LLM agents for education: Advances and applications. arXiv preprint arXiv:2503.11733. · arXiv
- Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shi- rong Ma, Peiyi Wang, Xiao Bi, and 1 others. 2025. Deepseek-R1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948. · arXiv
- Hongliang He, Wenlin Yao, Kaixin Ma, Wenhao Yu, Yong Dai, Hongming Zhang, Zhenzhong Lan, and Dong Yu. 2024a. Webvoyager: Building an end-to- end web agent with large multimodal models. In Proceedings of the 62nd Annual Meeting of the As- sociation for Computational Linguistics (Volume 1: Long Papers), pages 6864-6890.
- Wenyi Hong, Weihan Wang, Qingsong Lv, Jiazheng Xu, Wenmeng Yu, Junhui Ji, Yan Wang, Zihan Wang, Yuxiao Dong, Ming Ding, and 1 others. 2024. Coga- gent: A visual language model for gui agents. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14281-14290.
- PMLR.
- Kenton Lee, Mandar Joshi, Iulia Raluca Turc, Hexi- ang Hu, Fangyu Liu, Julian Martin Eisenschlos, Ur- vashi Khandelwal, Peter Shaw, Ming-Wei Chang, and Kristina Toutanova. 2023. Pix2struct: Screen- shot parsing as pretraining for visual language under- standing. In International Conference on Machine Learning, pages 18893-18912.
- PMLR.
- Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, and 1 others. 2021. WebGPT: Browser-assisted question-answering with human feedback. arXiv preprint arXiv:2112.09332. · arXiv
- Liangbo Ning, Ziran Liang, Zhuohang Jiang, Haohao Qu, Yujuan Ding, Wenqi Fan, Xiao-yong Wei, Shanru Lin, Hui Liu, Philip S Yu, and 1 others. 2025. A sur- vey of webagents: Towards next-generation ai agents for web automation with large foundation models. arXiv preprint arXiv:2503.23350. · arXiv
- Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. 2020. Zero: Memory optimizations toward training trillion parameter models. In SC20: International Conference for High Performance Com- puting, Networking, Storage and Analysis, pages 1- 16. IEEE.
- Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, and 1 others. 2024. Deepseek- math: Pushing the limits of mathematical reason- ing in open language models. arXiv preprint arXiv:2402.03300. · arXiv
- Peter Shaw, Mandar Joshi, James Cohan, Jonathan Be- rant, Panupong Pasupat, Hexiang Hu, Urvashi Khan- delwal, Kenton Lee, and Kristina N Toutanova. 2023. From pixels to ui actions: Learning to follow in- structions via graphical user interfaces. Advances in Neural Information Processing Systems, 36:34354- 34370.
- Mohit Shridhar, Jesse Thomason, Daniel Gordon, Yonatan Bisk, Winson Han, Roozbeh Mottaghi, Luke Zettlemoyer, and Dieter Fox. 2020. Alfred: A bench- mark for interpreting grounded instructions for ev- eryday tasks. In Proceedings of the IEEE/CVF con- ference on computer vision and pattern recognition, pages 10740-10749.
- Huatong Song, Jinhao Jiang, Yingqian Min, Jie Chen, Zhipeng Chen, Wayne Xin Zhao, Lei Fang, and Ji- Rong Wen. 2025. R1-searcher: Incentivizing the search capability in LLMs via reinforcement learning. arXiv preprint arXiv:2503.05592. · arXiv
- Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, and 1 others. 2025. Kimi k1. 5: Scaling reinforcement learning with llms. arXiv preprint arXiv:2501.12599. · arXiv
- Zora Zhiruo Wang, Jiayuan Mao, Daniel Fried, and Graham Neubig. 2024b. Agent workflow memory. arXiv preprint arXiv:2409.07429. · arXiv
- Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Beibin Li, Erkang Zhu, Li Jiang, Xiaoyun Zhang, Shaokun Zhang, Jiale Liu, Ahmed Hassan Awadallah, Ryen W White, Doug Burger, and Chi Wang. 2024. Autogen: Enabling next-gen LLM applications via multi-agent conversations. In First Conference on Language Modeling.
- Ke Yang, Yao Liu, Sapana Chaudhary, Rasool Fakoor, Pratik Chaudhari, George Karypis, and Huzefa Rang- wala. 2025b. AgentOccam: A simple yet strong base- line for LLM-based web agents. In The Thirteenth International Conference on Learning Representa- tions.
Cited by
Sorted by citations
RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement LearningPreprint
Wang, Z., Wang, K., Wang, Q., Zhang, P., Li, L., Yang, Z., Jin, X., Yu, K. F., Nguyen, M. N., Liu, L., et al. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2504.20073 · Cited by 3 · see citing works
“Stale or misaligned rollouts can induce collapse, while maintaining freshness—alongside limits on action granularity and sufficient task diversity—enables stable and effective RL training.”
Self-citationWebDancer: Towards Autonomous Information Seeking Agency
Wu, J.-L., Li, B., Fang, R.-N., et al. (2025) · Neural Information Processing Systems · doi:10.48550/arxiv.2505.22648 · Cited by 204
“The SFT stage offers strong initialization for the subsequent RL stage [29].”
The Surprising Effectiveness of Negative Reinforcement in LLM Reasoning
Zhu, X., Xia, M., Wei, Z., et al. (2025) · Neural Information Processing Systems · doi:10.48550/arxiv.2506.01347 · Cited by 181
“…scaling has emerged as a promising direction for enhancing model performance [2, 3, 4, 29, 35, 38, 42, 46, 49, 50, 51, 56, 59, 80, 67, 73, 78, 55]— particularly in reasoning tasks where generating multiple candidate solutions [7, 80] or longer reasoning traces [35, 68, 75] can help hit the…”
Self-citationLarge Language Models for Information Retrieval: A Survey
Zhu, Y., Yuan, H., Wang, S., Liu, J., Liu, W., Deng, C., Chen, H., Liu, Z., Dou, Z., & Wen, J.-R. (2026) · ACM Transactions on Information Systems · doi:10.1145/3748304 · Cited by 225 · see citing works
“Traditional evaluation metrics for ranking primarily focus on comparing the retrieval results of IR models with ground-truth (relevance) labels.”
One Token to Fool LLM-as-a-Judge
Zhao, Y., Liu, H., Yu, D., Kung, S., Chen, M., Mi, H., & Yu, D. (2025) · arXiv · doi:10.48550/arxiv.2507.08794
Hierarchy-of-Groups Policy Optimization for Long-Horizon Agentic Tasks
He, S., Feng, L., Wei, Q., et al. (2026) · arXiv.org · doi:10.48550/arxiv.2602.22817 · Cited by 52
From Web Search towards Agentic Deep Research: Incentivizing Search with Reasoning AgentsPreprint
Zhang, W., Li, Y., Bei, Y., Luo, J., Wan, G., Yang, L., Xie, C., Yang, Y., Huang, W., Miao, C., et al. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2506.18959
“Designing systems that facilitate user feedback enables continuous learning and adaptation, aligning AI outputs more closely with user expectations (Zhong et al., 2024; Zou et al., 2025c) and user-specific requirements (Zhang et al., 2025b).”
Self-citationAgentic Reinforcement Learning with Implicit Step Rewards
Liu, X., Wang, K., Wu, Y.-C., et al. (2025) · arXiv.org · doi:10.48550/arxiv.2509.19199 · Cited by 51
“For example, Zeng et al. (2025); Zou et al. (2025); Zhang et al. (2025b) provide denser feedback at intermediate steps but require handcrafted step labels that are costly, biased, and vulnerable to reward hacking.”
MobileGUI-RL: Advancing Mobile GUI Agent through Reinforcement Learning in Online EnvironmentPreprint
Shi, Y., Yu, W., Li, Z., Wang, Y., Zhang, H., Liu, N., Mi, H., & Yu, D. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2507.05720
“Similarly, WebAgent-R1 employed a multi-turn RL framework for web navigation, significantly boosting base model success rates by learning directly from online interactions with binary success signals (Wei et al., 2025).”
Harnessing Uncertainty: Entropy-Modulated Policy Gradients for Long-Horizon LLM Agents
Wang, J., Liu, J.-C., Fu, Y., et al. (2025) · arXiv.org · doi:10.48550/arxiv.2509.09265 · Cited by 50
“Recent research on RL-based agents, such as Search-R1 [13], SWE-RL [33], and WebAgent-R1 [34], has demonstrated that RL can effectively enhance agent performance and enable learning in highly interactive and dynamic environments.”
Scaling Agent Learning via Experience Synthesis
Chen, Z., Zhao, Z., Zhang, K., Liu, B., Qi, Q., Wu, Y., Kalluri, T., Cao, S., Xiong, Y., Tong, H., et al. (2025) · arXiv · doi:10.48550/arxiv.2511.03773
“Since the full evaluation set in WebArena is large and contains many similar tasks, we follow prior work (Qi et al., 2024; Wei et al., 2025) and evaluate agents on WebArena-Lite (Liu et al., 2024), a more balanced subset of 165 high-quality, challenging tasks selected from the original 812.”
PersonaAgent: When Large Language Model Agents Meet Personalization at Test Time
Zhang, W., Zhang, X.-Y., Zhang, C.-W., Yang, L., Shang, J., Wei, Z., Zou, H. P., Huang, Z., Wang, Z., Gao, Y.-F., et al. (2025) · arXiv.org · doi:10.48550/arxiv.2506.06254 · Cited by 42
“…LLMs, LLM-empowered agents (Luo et al., 2025) represent a paradigm shift, integrating external tools (Qin et al., 2024; Yuan et al., 2024; Wei et al., 2025), memory mechanisms (Hatalis et al., 2023; Zhong et al., 2024), and goal-directed reasoning (Yao et al., 2023b,a) to enhance their…”
Self-citationARE: Scaling Up Agent Environments and EvaluationsPreprint
Andrews, P., Benhalloum, A., Bertran, G. M.-T., et al. (2025) · arXiv.org · doi:10.48550/arxiv.2509.17158 · Cited by 33 · see citing works
“Verifiable rewards have proven crucial for improving reasoning (DeepSeek-AI et al., 2025; Lambert et al., 2024), code generation (Gehring et al., 2024), agentic web browsing (OpenAI, 2025a; Wei et al., 2025b) and software engineering (Yang et al., 2025b; MoonshotAI et al., 2025).”
ExpSeek: Self-Triggered Experience Seeking for Web Agents
Zhang, W., Zhang, X., Yu, H., et al. (2026) · Annual Meeting of the Association for Computational Linguistics · doi:10.48550/arxiv.2601.08605 · Cited by 30
Self-citationSFR-DeepResearch: Towards Effective Reinforcement Learning for Autonomously Reasoning Single AgentsPreprint
Nguyen, X.-P., Pandit, S., Reddy, R. G., Xu, A., Savarese, S., Xiong, C., & Joty, S. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2509.06283
“For web agents, cold-start instruction-tuning followed by RL with customized objectives and roll-out strategies is commonly employed [39, 17, 31, 42, 26].”
SelfElicit: Your Language Model Secretly Knows Where is the Relevant Evidence
Liu, Z., Amjad, R. A., Adkathimar, R., et al. (2025) · Annual Meeting of the Association for Computational Linguistics · doi:10.48550/arxiv.2502.08767 · Cited by 30
Lin, M.-H., Wu, Z., Xu, Z.-C., et al. (2025) · arXiv.org · doi:10.48550/arxiv.2510.16724 · Cited by 29
“, and it will return the top searched results between <information> and </information> .”
Customer-R1: Personalized Simulation of Human Behaviors via RL-based LLM Agent in Online Shopping
Wang, Z., Lu, Y., Zhang, Y., et al. (2025) · arXiv.org · doi:10.48550/arxiv.2510.07230 · Cited by 25
“Wei et al. (2025) propose an end-to-end multi-turn RL framework for web agents, achieving higher task success rates by optimizing agent decisions over long-horizon interactions.”
Atom-Searcher: Enhancing Agentic Deep Research via Fine-Grained Atomic Thought RewardPreprint
Deng, Y., Wang, G., Ying, Z., Wu, X., Lin, J., Xiong, W., Dai, Y., Yang, S., Zhang, Z., Wang, Q., et al. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2508.12800 · Cited by 1 · see citing works
Verified Critical Step Optimization for LLM Agents
Li, M.-K., Zeng, Q., Fang, T., Liang, Z.-W., Song, L.-F., Liu, Q., Mi, H.-T., & Yu, D. (2026) · Annual Meeting of the Association for Computational Linguistics · doi:10.48550/arxiv.2602.03412 · Cited by 28
A Survey on Agentic Multimodal Large Language ModelsPreprint
Yao, H., Zhang, R., Huang, J., Zhang, J., Wang, Y., Fang, B., Zhu, R., Jing, Y., Liu, S., Li, G., et al. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2510.10991 · Cited by 1 · see citing works
“They demonstrate remarkable capability in automating complex digital tasks across diverse software environments and operating systems, including web scenarios [122], mobile platforms [407], and desk-top interfaces [157, 408].”
WebXSkill: Skill Learning for Autonomous Web Agents
Wang, Z., Wu, Q., Zhang, X.-C., et al. (2026) · arXiv.org · doi:10.48550/arxiv.2604.13318 · Cited by 20
Self-citationHu, M., Fang, T., Zhang, J., Ma, J., Zhang, Z., Zhou, J., Zhang, H., Mi, H., Yu, D., & King, I. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2505.20013
“While directly applying Reinforcement Learning (RL) in agentic environments (Qi et al., 2025; Li et al., 2025; Liu et al., 2025; Singh et al., 2025; Wei et al., 2025) is a viable alternative, the resulting reasoning abilities are often unpredictable and lack structured priors.”
The Confidence Dichotomy: Analyzing and Mitigating Miscalibration in Tool-Use Agents
Xuan, W., Zeng, Q., Qi, H.-L., et al. (2026) · Annual Meeting of the Association for Computational Linguistics · doi:10.48550/arxiv.2601.07264 · Cited by 19
Lu, Z.-M., Ren, H., Yang, Y.-Q., et al. (2025) · arXiv.org · doi:10.48550/arxiv.2509.22644 · Cited by 21
“In the field of code agents, various works (Pan et al., 2025; Yang et al., 2025b; Zhang et al., 2025b; Wang et al., 2025a; Ma et al., 2024; Xie et al., 2025; Jain et al., 2025; Guo et al., 2025c; Ma et al., 2025a) leverage supervised fine-tuning combined with software engineering data synthesis and…”
UItron: Foundational GUI Agent with Advanced Perception and PlanningPreprint
Zeng, Z., Huang, J., Zheng, L., Han, W., Zhong, Y., Chen, L., Yang, L., Chu, Y., He, Y., & Ma, L. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2508.21767
“Further optimizations have focused on issues such as reasoning patterns [41, 86], sparse rewards [80, 54], and online learning [54, 65].”
SAGE-OPD: Selective Agent-Guided Intervention for Multi-Turn On-Policy Distillation
Zhou, Y.-H., Zhang, L.-Z., Wu, Y.-F., et al. (2026) · arXiv.org · doi:10.48550/arxiv.2606.19659 · Cited by 17
“The second is reinforcement learning with verifiable rewards (RLVR) (Shao et al., 2024; Zhou et al., 2025b; Yang et al., 2025b), where the agent receives reward from the final environment outcome (Luo et al., 2025; Wei et al., 2025).”
Autonomous Agents for Scientific Discovery: Orchestrating Scientists, Language, Code, and PhysicsPreprint
Zhou, L., Ling, H., Fu, C., Huang, Y.-P., Sun, M., Yu, W., Wang, X.-X., Li, X.-E., Su, X.-Y., Zhang, J., et al. (2025) · arXiv.org · doi:10.48550/arxiv.2510.09901 · Cited by 20 · see citing works
ReSkill: Reconciling Skill Creation with Policy Optimization in Agentic RLPreprint
He, Z.-L., Lin, H.-T., Han, B.-R., Zhu, W.-Y., Fang, H.-Y., Wang, Y.-Y., Zhu, X.-C., Li, R.-Z., & Reimherr, M. (2026) · arXiv.org · doi:10.48550/arxiv.2606.01619 · Cited by 19 · see citing works
“…reinforcement learning (RL), where agents are trained from verifiable environment rewards, has emerged as a promising paradigm (Jin et al., 2025; Wei et al., 2025; Da et al., 2025), yet the resulting policies do not systematically accumulate transferable problem-solving strategies across tasks.”
OpenEarthAgent: A Unified Framework for Tool-Augmented Geospatial Agents
Shabbir, A., Sheikh, M. U., Munir, M. A., et al. (2026) · arXiv.org · doi:10.48550/arxiv.2602.17665 · Cited by 16
“Subsequent systems, including We-bAgent [42], VisTA [14], and DeepEyes [50], introduced modular architectures capable of selecting and sequencing tools under guided supervision, improving reasoning depth and execution consistency.”
DraCo: Draft as CoT for Text-to-Image Preview and Rare Concept Generation
Jiang, D., Zhang, R., Li, H., Zong, Z., Guo, Z., He, J., Guo, C., Ye, J., Fang, R., Li, W., et al. (2026) · 2026 IEEE/CVF Conference on Computer Vision and Pattern Recognition- FINDINGS Track (CVPRF) · doi:10.1109/cvprf71909.2026.00567
“…apples” “two cats on the left and three backpacks on the right” soning [24, 58] have shown remarkable success across various domains, including mathematical problems solving [1, 7, 17, 31, 72, 73], visual reasoning [4, 12–14, 22, 36, 48, 67, 70, 71], and multi-agent systems [16, 32, 33, 35, 59].”
WebSynthesis: World-Model-Guided MCTS for Efficient WebUI-Trajectory Synthesis
Gao, Y., Ye, J., Wang, J., et al. (2025) · arXiv.org · doi:10.48550/arxiv.2507.04370 · Cited by 16
“Additionally, directly training agents via live web interactions is often unpredictable and costly due to noise from real-world environments and the substantial number of API calls required, making algorithm like reinforcement learning (RL) prohibitively expensive [18, 13, 23].”
Abstain-R1: Calibrated Abstention and Post-Refusal Clarification via Verifiable RL
Zhai, S., Liang, J., & Kang, D. (2026) · Annual Meeting of the Association for Computational Linguistics · doi:10.48550/arxiv.2604.17073 · Cited by 15
Rank-GRPO: Training LLM-based Conversational Recommender Systems with Reinforcement Learning
Zhu, Y.-C., Steck, H., Liang, D., et al. (2025) · arXiv.org · doi:10.48550/arxiv.2510.20150 · Cited by 15
DynaWeb: Model-Based Reinforcement Learning of Web Agents
Ding, H., Liu, P.-D., Wang, J.-Q., et al. (2026) · arXiv.org · doi:10.48550/arxiv.2601.22149 · Cited by 14
“A representative end-to-end online RL approach is WebAgent-R1 [Wei et al., 2025], which optimizes multi-turn web interaction policies using outcome-based rewards and scalable trajectory sampling (e.g., multi-group GRPO [Shao et al., 2024]).”
InfluentialWebCoach: Self-Evolving Web Agents with Cross-Session Memory Guidance
Liu, G., Geng, S.-J., Li, S., et al. (2025) · arXiv.org · doi:10.48550/arxiv.2511.12997 · Cited by 14
“…hierarchical planning, improving performance on benchmarks like WebArena and WebShop while underscoring the importance of strong visual grounding (Wei et al., 2025; Zhang et al., 2025d; Gandhi & Neubig, 2025; Li et al., 2025; Yang et al., 2024; Putta et al., 2024; Zheng et al., 2024; Nakano et…”
From Question Answering to Task Completion: A Survey on Agent System and Harness DesignPreprint
Guo, J.-Y., Hao, Z.-W., Wang, C.-C., Fan, C., Luo, T.-Z., Li, H.-G., Gao, Y., Mei, H., Peng, J., Xu, R.-J., et al. (2026) · arXiv.org · doi:10.48550/arxiv.2606.20683 · Cited by 15 · see citing works
“Rather than relying solely on prompts, workflows, or run-time orchestration, models are increasingly trained to plan, use tools, verify intermediate states, and recover from errors in interactive environments [42], [152]–[156].”
When Users Change Their Mind: Evaluating Interruptible Agents in Long-Horizon Web Navigation
Zou, H. P., Miao, C., Huang, W.-C., Chen, Y., Zhou, Y., Zhang, H., Wu, Y., Fang, L., Gu, Z., Zhang, Z., et al. (2026) · arXiv · doi:10.48550/arxiv.2604.00892
“WebArena-Lite (Koh et al., 2024; Wei et al., 2025) is a curated version of WebArena designed for more reliable evaluation.”
InfluentialSelf-citationGTM: Simulating the World of Tools for AI Agents
Ren, Z.-Z., Zhang, X.-P., Qian, Z.-X., Gao, Y., Shi, Y., Zheng, S.-X., & He, J. (2025) · arXiv.org · doi:10.48550/arxiv.2512.04535 · Cited by 13
“DeepResearcher (Zheng et al., 2025) and WebAgent-R1 (Wei et al., 2025) exemplify this approach, embedding tools within iterative workflows where agents learn optimal strategies through trial-and-error.”
SynthAgent: Adapting Web Agents with Synthetic Supervision
Wang, Z.-Y., Liang, Y.-M., Zhang, X.-C., et al. (2025) · Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2026.acl-long.716 · Cited by 13
“Another series of works (Lu et al., 2025; Qiu et al., 2025; Wei et al., 2025b) use reinforcement learning to train web agents, but they often require supervision from annotation and are impractical for complex and realistic websites due to the cost of online training.”
Unlocking the Power of Multi-Agent LLM for Reasoning: From Lazy Agents to Deliberation
Zhang, Z.-W., Li, X.-M., Lin, Y.-D., Liu, H., Chandradevan, R., Wu, L.-L., Lin, M., Wang, F., Tang, X.-F., He, Q., et al. (2025) · International Conference on Learning Representations · doi:10.48550/arxiv.2511.02303 · Cited by 15
A Subgoal-driven Framework for Improving Long-Horizon LLM Agents
Wang, T., Gooding, S., Hartmann, F., et al. (2026) · arXiv.org · doi:10.48550/arxiv.2603.19685 · Cited by 12
OpenApps: Simulating Environment Variations to Measure UI-Agent Reliability
Ullrich, K., Su, J., Shi, C., et al. (2025) · arXiv.org · doi:10.48550/arxiv.2511.20766 · Cited by 12
“WebAgent-R1 (Wei et al., 2025) investigates end-to-end, multi-turn RL for agents.”
EviNote-RAG: Enhancing RAG Models via Answer-Supportive Evidence Notes
Dai, Y.-Q., Wang, G., Wang, Y., et al. (2025) · arXiv.org · doi:10.48550/arxiv.2509.00877 · Cited by 12
“Subsequent works (Qi et al., 2025; Chen et al., 2025; Jin et al., 2025) broadened this paradigm by exploring diverse scenarios and tool-use (Wei et al., 2025b; Dai et al., 2025), further underscoring the performance gains enabled by reward-based supervision for retrieval-augmented reasoning.”
GUI-Eyes: Tool-Augmented Perception for Visual Grounding in GUI Agents
Chen, C., Shao, J.-W., Lu, D., et al. (2026) · AAAI Conference on Artificial Intelligence · doi:10.48550/arxiv.2601.09770 · Cited by 11
“This idea has been extended to GUI agents (Lu et al. 2025; Luo et al. 2025; Zhou et al. 2025; Liu et al. 2025b; Gao, Zhang, and Xu 2025; Wei et al. 2025; Gu et al. 2025), showing strong data efficiency.”
Memory-R2: Fair Credit Assignment for Long-Horizon Memory-Augmented LLM AgentsPreprint
Yan, S., Bahloul, A., Nie, E., Schwarzmann, S., Trivisonno, R., Tresp, V., & Ma, Y. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2605.21768
“Reinforcement learning has recently become an effective paradigm for training LLM agents in interactive settings such as tool use, web navigation, and reasoning [12, 4, 18, 2].”
Android Coach: Improve Online Agentic Training Efficiency with Single State Multiple Actions
Gan, G., Ding, Y., Chen, C., et al. (2026) · Annual Meeting of the Association for Computational Linguistics · doi:10.48550/arxiv.2604.07277 · Cited by 10
UI-Voyager: A Self-Evolving GUI Agent Learning via Failed ExperiencePreprint
Lin, Z.-C., Liu, F., Yang, Y., et al. (2026) · arXiv.org · doi:10.48550/arxiv.2603.24533 · Cited by 11 · see citing works
“…models (LLMs) and large vision-language models (LVLMs) have led to significant breakthroughs in intent comprehension, multi-modal reasoning, and GUI understanding (Wei et al., 2022b; You et al., 2024; Li et al., 2024; Hong et al., 2024; Zhang et al., 2025c; Lin et al., 2025a; Liu et al., 2026).”
Self-citationChen, X., Zhang, Z.-Q., Guo, Y.-W., et al. (2026) · Annual Meeting of the Association for Computational Linguistics · doi:10.48550/arxiv.2603.04783 · Cited by 9
“Yet, multi-turn interaction has emerged as the ubiquitous paradigm for human-AI engagement, serving as the backbone for applications ranging from general-purpose chat systems (OpenAI, 2025a; Google, 2025) to complex agentic workflows (Liu et al., 2025; Wei et al., 2025).”
OpAgent: Operator Agent for Web Navigation
Guo, Y., Yang, W., Yang, S.-Y., et al. (2026) · arXiv.org · doi:10.48550/arxiv.2602.13559 · Cited by 9
“To bridge the gap between training and execution, recent research has begun exploring Online RL in simulated environments with text observation Wei et al. (2025).”
Lin, M.-H., Dai, E., Liu, H., et al. (2026) · arXiv.org · doi:10.48550/arxiv.2602.00528 · Cited by 9
Dyna-Mind: Learning to Simulate from Experience for Better AI Agents
Yu, X., Peng, B., Galley, M., et al. (2025) · arXiv.org · doi:10.48550/arxiv.2510.09577 · Cited by 9
Escaping the Context Bottleneck: Active Context Curation for LLM Agents via Reinforcement LearningPreprint
Li, X., Lyu, T., Yang, Y., Shan, L., Yang, S., Zhang, L., Huang, Z., Liu, Q., & Li, Y. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2604.11462 · Cited by 1 · see citing works
“RL is a pivotal paradigm for aligning agentic reasoning across diverse domains (e.g., web navigation, embodied environments) using algorithms like PPO and GRPO (Schulman et al., 2017; Shao et al., 2024; Yao et al., 2022; Wei et al., 2025; Jin et al., 2025b).”
Multi-Turn Reinforcement Learning for Tool-Calling Agents with Iterative Reward CalibrationPreprint
Modecrua, W., Kaewtawee, K., Pachtrachai, K., & Kraisingkorn, T. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2604.02869
“WebAgent-R1 (Wei et al., 2025) finds sparse outcome rewards outperform dense rewards for web navigation, consistent with our naïve dense reward findings.”
InfluentialSQL-Trail: Multi-Turn Reinforcement Learning with Interleaved Feedback for Text-to-SQL
Hua, H., Han, Z., Shen, Z., et al. (2026) · Annual Meeting of the Association for Computational Linguistics · doi:10.48550/arxiv.2601.17699 · Cited by 8
“Recent work on tool-augmented RL has shown that LLM agents can improve substantially by interacting with external environments over multiple turns (e.g., search, UI, code execution) rather than acting in a single pass (Jin et al., 2025; Wei et al., 2025a; Hu et al., 2025; Wei et al., 2025b).”
Li, C.-L., Elmahdy, A., Boyd, A., et al. (2025) · doi:10.48550/arxiv.2511.20718 · Cited by 8
“By addressing both the granularity mismatch in credit assignment and the instability induced by increasingly off-policy updates, our approach enables more stable and conservative optimization.”
Xu, A., Nguyen, X.-P., Zhou, Y., Wu, C.-S., Xiong, C., & Joty, S. R. (2025) · arXiv.org · doi:10.48550/arxiv.2510.17793 · Cited by 8
“Agentic settings introduce additional wrinkles: With autonomously acting single agents (OpenAI, 2025; Nguyen et al., 2025; Wei et al., 2025) and complex multi-agent workflows (Liang et al., 2025; Alzubi et al., 2025) now being set free to browse the web and act on behalf of users with minimal…”
CoCo: Code as CoT for Text-to-Image Preview and Rare Concept Generation
Li, H.-D., Qing, C.-M., Zhang, H., et al. (2026) · arXiv.org · doi:10.48550/arxiv.2603.08652 · Cited by 7
“Concurrently, chain-of-thought (CoT) reasoning [16, 17] has demonstrated significant improvements across a broad spectrum of tasks, spanning mathematical problem solving [18–21], visual reasoning [22–28], and multi-agent systems [29, 30].”
WebFactory: Automated Compression of Foundational Language Intelligence into Grounded Web Agents
Fan, S.-C., Shi, Q., Xu, S.-Z., et al. (2026) · arXiv.org · doi:10.48550/arxiv.2603.05044 · Cited by 7
“…innovations include curricula derived from agent failure modes and outcome-supervised reward modeling (Qi et al., 2024), success-driven rollouts (Wei et al., 2025), reusable skill abstractions (Zheng et al., 2025), and hierarchical formulations for decomposing complex browsing workflows into…”
Dai, Y.-Q., Yang, S., Wang, G., et al. (2025) · AAAI Conference on Artificial Intelligence · doi:10.48550/arxiv.2508.07956 · Cited by 7
“Zheng et al. 2025; Wei et al. 2025) has started exploring RAG in web-based environments, showing the benefits of using web search during model training.”
Yan, S., Yang, X., Huang, Z., Nie, E., Ding, Z., Li, Z., Ma, X., Bi, J., Kersting, K., Pan, J. Z., et al. (2026) · Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2026.acl-long.583 · Cited by 10 · see citing works
Li, Y.-F., Zhou, Q., Duan, W.-J., et al. (2026) · arXiv.org · doi:10.48550/arxiv.2606.05885 · Cited by 6
“These capabilities pose significant challenges for reinforcement learning (RL), particularly in assigning credit to intermediate decisions in long-horizon trajectories (Pignatelli et al., 2023; Wei et al., 2025).”
SEARL: Joint Optimization of Policy and Tool Graph Memory for Self-Evolving Agents
Feng, X.-S., Song, X., Li, L., et al. (2026) · Annual Meeting of the Association for Computational Linguistics · doi:10.48550/arxiv.2604.07791 · Cited by 6
“Reinforcement learning has become a central paradigm for improving agent decision-making in multi-turn environments, addressing challenges like long-horizon credit assignment and sparse rewards (Wu et al., 2025b; Mialon et al., 2023; Wei et al., 2025; Zhuang et al., 2025; Xu et al., 2026).”
Self-citationReRec: Reasoning-Augmented LLM-based Recommendation Assistant via Reinforcement Fine-tuning
Huang, J.-N., Wang, S., Ning, L.-B., et al. (2026) · Annual Meeting of the Association for Computational Linguistics · doi:10.48550/arxiv.2604.07851 · Cited by 6
“Recent studies often use rule-based rewards in RFT to improve LLM reasoning capabilities (Jin et al., 2025; Huang et al., 2025b; Wei et al., 2025; Luo et al., 2025).”
SAGE: Multi-Agent Self-Evolution for LLM ReasoningPreprint
Peng, Y., Zhu, X., Wei, C., Zeng, N., Wang, L., He, Y., & Yu, F. R. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2603.15255
“WebAgent-R1 (Wei et al., 2025) is an end-to-end multi-turn RL framework that significantly boosts web navigation success using binary success re-wards.”
Generalization in Online Reinforcement Learning for Mobile AgentsPreprint
Gu, L., Jiang, Z., Chi, Z., Liu, H., Wang, Z., Yu, Y., Berseth, G., & Wang, Y. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2603.07432
“More recent studies bring RL into realistic GUI-based environments, particularly in web and computer-use domains (Qi et al., 2025; Wei et al., 2025; Wu et al., 2025a; Vattikonda et al., 2025; Lu et al., 2025; Feng et al., 2025).”
LongNav-R1: Horizon-Adaptive Multi-Turn RL for Long-Horizon VLA Navigation
Hu, Y., Xi, A., Xiao, Q.-X., et al. (2026) · Robotics · doi:10.48550/arxiv.2602.12351 · Cited by 6
“While these advancements have primarily been applied to web search [49, 53] and tool-use [46, 54, 47], adapting these benefits to embodied navigation remains an open challenge.”
WebArbiter: A Principle-Guided Reasoning Process Reward Model for Web Agents
Zhang, Y., Tang, S., Li, Z., et al. (2026) · arXiv.org · doi:10.48550/arxiv.2601.21872 · Cited by 6
EntWorld: A Holistic Environment and Benchmark for Verifiable Enterprise GUI Agents
Mo, Y., Bai, Y., Sun, D.-P., et al. (2026) · arXiv.org · doi:10.48550/arxiv.2601.17722 · Cited by 6
“We adopt GRPO as our reinforcement learning algorithm to train the multimodal agents through end-to-end multi-turn interactions following the prior works [Qi et al., 2025, Wei et al., 2025].”
ColorBrowserAgent: Complex Long-Horizon Browser Agent with Adaptive Knowledge Evolution
Wang, J.-H., Zhou, J., Zhang, W., et al. (2026) · Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2026.acl-industry.46 · Cited by 6
“Existing approaches (Wei et al., 2025; Qi et al., 2024) adapt to specific web environments by internalizing site-specific behaviors into model parameters through additional training.”
Self-citationMTSQL-R1: Towards Long-Horizon Multi-Turn Text-to-SQL via Agentic Training
Guo, T., Wang, H., Liu, C., et al. (2025) · Annual Meeting of the Association for Computational Linguistics · doi:10.48550/arxiv.2510.12831 · Cited by 6
“Models like Search-R1 (Jin et al., 2025) and WebAgent-R1 (Wei et al., 2025) extend reasoning via environment interaction.”
Beyond Outcome Reward: Decoupling Search and Answering Improves LLM Agents
Wang, Y., Wei, Z., Zhu, X.-Y., et al. (2025) · arXiv.org · doi:10.48550/arxiv.2510.04695 · Cited by 6
“…inherent factuality limitations of large language models (LLMs), including knowledge cut-offs (Cheng et al., 2024) and hallucinations (Huang et al., 2025), have been increasingly mitigated by leveraging external tools like retrieval systems (Lewis et al., 2020) and web search (Wei et al., 2025c).”
Self-citationSymbolic Graphics Programming with Large Language Models
Chen, Y., Zhang, H.-Q., Huang, Y.-Y., et al. (2025) · Trans. Mach. Learn. Res. · doi:10.48550/arxiv.2509.05208 · Cited by 6
“Response format:”
LOTAPO: Leave-One-Turn Attribution for Self-Generated Process Rewards in Multi-Turn Search Reasoning
Zhu, Q., Wu, J.-J., & Wang, L. (2026) · arXiv.org · doi:10.48550/arxiv.2607.13501 · Cited by 5
EvoCUA-1.5: Online Reinforcement Learning for Multi-turn Computer-Use Agents
Huang, M.-Q., Xue, T., Peng, C., et al. (2026) · arXiv.org · doi:10.48550/arxiv.2607.09773 · Cited by 5
OpenWebRL: Demystifying Online Multi-turn Reinforcement Learning for Visual Web Agents
Yang, R., Wu, Q., Chen, Y.-X., et al. (2026) · arXiv.org · doi:10.48550/arxiv.2606.02031 · Cited by 6
“Multi-turn GRPO [46] propagates this trajectory-level signal to the action tokens generated across all turns in τ i .”
InfluentialSelf-citationSIRI: Self-Internalizing Reinforcement Learning with Intrinsic Skills for LLM Agent Training
He, Z., Li, Y.-F., Huang, F., et al. (2026) · arXiv.org · doi:10.48550/arxiv.2606.02355 · Cited by 5
“Other work explores hierarchical training, online tuning, end-to-end web-agent RL, hindsight credit, and retrieval-augmented exploration (Zhou et al., 2024; Putta et al., 2024; Feng et al., 2025a; Chen et al., 2025; Wei et al., 2025; Tan et al., 2026; Zhang et al., 2026b).”
WebForge: Breaking the Realism-Reproducibility-Scalability Trilemma in Browser Agent Benchmark
Yuan, P., Yin, Y.-Y., Cai, Y., et al. (2026) · arXiv.org · doi:10.48550/arxiv.2604.10988 · Cited by 5
“Autonomous web agents that navigate websites, fill forms, and complete multi-step tasks have improved rapidly [6,11,31,33,38,40].”
Self-citationEvoFSM: Controllable Self-Evolution for Deep Research with Finite State Machines
Zhang, S., Yuan, C., Guo, R., et al. (2026) · arXiv.org · doi:10.48550/arxiv.2601.09465 · Cited by 5
“However, dominant approaches still rely heavily on pre-defined, static procedural paradigms, typically characterized by fixed, iterative “toolcall-generation-reflection” pipelines (Wei et al., 2025; Zhao et al., 2025).”
Towards Agentic Self-Learning LLMs in Search Environment
Sun, W., Cheng, X., Fan, J., et al. (2025) · arXiv.org · doi:10.48550/arxiv.2510.14253 · Cited by 5
DeepPlanner: Scaling Planning Capability for Deep Research Agents via Advantage Shaping
Fan, W., Yao, W.-L., Li, Z., et al. (2025) · Annual Meeting of the Association for Computational Linguistics · doi:10.48550/arxiv.2510.12979 · Cited by 5
“…research capability, as demonstrated by ReSearch (Chen et al., 2025), R1-Searcher (Song et al., 2025), Search-R1 (Jin et al., 2025), WebRL (Qi et al., 2025), WebThinker (Li et al., 2025b), WebAgent-RL (Wei et al., 2025b), DeepResearcher (Zheng et al., 2025), and EvolveSearch (Zhang et al., 2025).”
Self-citationJEF-Hinter: Leveraging Offline Knowledge for Improving Web Agents Adaptation
Nekoei, H., Jaiswal, A., Bechard, P., et al. (2025) · doi:10.48550/arxiv.2510.04373 · Cited by 5
“LLMs have shown strong reasoning capabilities (Wei et al., 2022), resulting in LLM-based agents applied on a variety of real-world interactive tasks, including web navigation (Nakano et al., 2021; Wei et al., 2025; Zhang et al., 2025).”
Language Models can Self-Improve at State-Value Estimation for Better Search
Mendes, E., & Ritter, A. (2025) · Neural Information Processing Systems · doi:10.48550/arxiv.2503.02878 · Cited by 5
“Other work has trained agents from tree search-generated data [18, 47, 70] or with explicit reinforcement learning [46, 57, 14], but these methods usually require ground truth reward.”
Agentic-DPO: From Imitation to Agentic Policy Optimization on Expert Trajectories
Chen, Y.-X., & Yuille, A. (2026) · arXiv.org · doi:10.48550/arxiv.2607.10601 · Cited by 4
“RLHF-style online RL [24, 30, 27, 3, 40, 39, 42] trains agents through environment rollouts on benchmarks such as WebShop [44], ALFWorld [35], and SWE-bench [16], requiring reward design and rollout workers; PivotRL [47] reduces this cost via informative pivot states.”
WARD: Adversarially Robust Defense of Web Agents Against Prompt InjectionsPreprint
Cao, T., Chen, Y., Cao, H., et al. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2605.15030
“Web agents [46, 83, 65, 70] are autonomous systems that execute user goals through web interactions such as clicking, typing, and browsing.”
Self-citationT2PO: Uncertainty-Guided Exploration Control for Stable Multi-Turn Agentic Reinforcement Learning
Wang, H.-X., Cui, H.-J., Zhang, C.-W., et al. (2026) · arXiv.org · doi:10.48550/arxiv.2605.02178 · Cited by 4
“The remaining high-scoring trajectories are treated as supervised demonstration data, from which we extract state–action pairs and perform one epoch of supervised fine-tuning on the policy model π θ as follows: This RFT stage equips the agent with a reliable initial policy that produces structurally valid actions and…”
InfluentialSelf-citationGUI Agents with Reinforcement Learning: Toward Digital InhabitantsPreprint
Hu, J., (33711), J. L., Lai, J., Hu, J., Sheng, Y., Chen, S., Li, J., Du, D., & Guo, S. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2604.27955 · Cited by 1 · see citing works
WebAgentGuard: A Reasoning-Driven Guard Model for Detecting Prompt Injection Attacks in Web AgentsPreprint
Chen, Y., Cao, T., Li, H., et al. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2604.12284
“Web agents are AI systems designed to automate time-consuming online tasks by interacting with web environments, leveraging vision–language models (VLMs) to interpret both visual and textual webpage content (Ning et al., 2025; Zhou et al., 2023; Wei et al., 2025; Wu et al., 2025).”
Self-citationStressWeb: A Diagnostic Benchmark for Web Agent Robustness under Realistic Interaction VariabilityPreprint
Bai, H., Wang, D., Chen, L., Hao, B., Shao, P., Yang, Y., He, Y., & Zhuang, C. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2604.16385
“WebAgent-R1 (Wei et al., 2025b) leverages end-to-end multi-turn reinforcement learning in simulated web environments to enhance agent recovery behaviors.”
Autonomous Continual Learning for Environment Adaptation of Computer-Use Agents
Xue, T., Liao, Z., Shi, T.-N., et al. (2026) · doi:10.48550/arxiv.2602.10356 · Cited by 4
“Existing work has explored a wide range of interaction paradigms, spanning from DOM-based web manipulation (Zhou et al., 2024; Deng et al., 2023; Wei et al., 2025c; Qi et al., 2025) to pixel-level control in desktop environments (Anthropic, 2024; 2025; Qin et al., 2025; Liao et al., 2025).”
Knowledge is Not Enough: Injecting RL Skills for Continual Adaptation
Tang, P., Wang, Y., & Zhang, M. (2026) · Annual Meeting of the Association for Computational Linguistics · doi:10.48550/arxiv.2601.11258 · Cited by 4
“Beyond single-turn reasoning, end-to-end RL is increasingly explored for training agentic LLMs that must plan over multi-turn interactions in external environments, including web search (Wei et al., 2025) and tool-use agents (Qian et al., 2025).”
WEBSERV: A Full-Stack and RL-Ready Web Environment for Training Web Agents at Scale
Lu, Y., Wang, Z., Huang, J., et al. (2025) · doi:10.48550/arxiv.2510.16252 · Cited by 4
“While early approaches relied on prompting [14, 10] or supervised fine-tuning [7], recent work has turned to Reinforcement Learning (RL) to enable self-improvement through interaction [15, 12, 3].”
From Storage to Experience: A Survey on the Evolution of LLM Agent Memory Mechanisms
Luo, J., Tian, Y., Cao, C., Luo, Z., Lin, H., Li, K., Kong, C., Yang, R., & Ma, J. (2026) · Findings of the Association for Computational Linguistics: ACL 2026 · doi:10.18653/v1/2026.findings-acl.2069 · Cited by 3 · see citing works
“…based on curricula facilitate exploration tasks of increasing difficulty through the dynamic generation and adjustment of sequences for tasks (Wei et al., 2025c; Ahn et al., 2022); and drivers based on reuse enable exploration of high efficiency through the abstraction and reuse of…”
SearchMaster: Grounded and Regulated Self-Play for Search Agents
Tan, W.-T., Cao, Q., Wang, J.-Q., et al. (2026) · doi:10.48550/arxiv.2608.01822 · Cited by 4
“LLM-based search agents improve knowledge-intensive question answering by retrieving evidence from the web (Nakano et al. 2021; Yao et al. 2022; Li et al. 2026a; Wei et al. 2025b).”
DocArena: Turning Raw Documents into Controllable Training Environments for Document Search Agents
Wang, J., Zhang, R., Yu, T., et al. (2026) · arXiv.org · doi:10.48550/arxiv.2606.26122 · Cited by 3
“Previous works advance search agents from different aspects, e.g. , improving reward modeling with progress rewards [32] or trajectory-level credit assignment [42,66], and enhancing search tools such as web search APIs [28,57] or document parsers.”
ScaleWoB: Guiding GUI Agents with Coding Agents via Large-Scale Environmental Synthesis
Liu, G., Ye, J.-L., Gao, P.-Z., et al. (2026) · doi:10.48550/arxiv.2605.25160 · Cited by 3
Self-citationMem-π: Adaptive Memory through Learning When and What to GeneratePreprint
Wang, X., Wang, C., Nekoei, H., Pal, C., Lacoste, A., Gella, S., Liu, B., & Taslakian, P. (2026) · arXiv.org · doi:10.48550/arxiv.2605.21463 · Cited by 3 · see citing works
Structured Distillation of Web Agent Capabilities Enables GeneralizationPreprint
Lù, X. H., & Reddy, S. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2604.07776
“We omit WebRL (Qi et al., 2024) and WebAgent-R1 (Wei et al., 2025) because they report on WebArena-Lite (165 tasks), not the full 812-task benchmark.”
Li, L., Zhou, Z.-J., Long, J.-X., et al. (2026) · Annual Meeting of the Association for Computational Linguistics · doi:10.48550/arxiv.2603.16161 · Cited by 3
Self-citationLiu, S.-Y., Yuan, H., Li, X., et al. (2026) · arXiv.org · doi:10.48550/arxiv.2601.09503 · Cited by 3
- Authors
- Wei, Z., Yao, W., Liu, Y., Zhang, W., Lu, Q., Qiu, L., Yu, C., Xu, P., Zhang, C., Yin, B., Yun, H., & Li, L.
- Affiliations
- Georgia Institute of Technology (US)
- University of Virginia (US)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2025
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2505.16421
- OpenAlex
- W4416451824
- Semantic Scholar
- a343a24a0d47e0acf4d71bfd0215924bc76d0256
- Cited by
- 0
- References
- 64
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Information Systems › Web Data Mining and Analysis
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
- Keywords
- web agents, multi-turn reinforcement learning, large language models, end-to-end learning, long-horizon decision-making, chain-of-thought reasoning, behavior cloning, warm-up, test-time scaling
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
