Autonomous Evaluation and Refinement of Digital Agents
Open accessAbstract
We show that domain-general automatic evaluators can significantly improve the performance of agents for web navigation and device control. We experiment with multiple evaluation models that trade off between inference cost, modularity of design, and accuracy. We validate the performance of these models in several popular benchmarks for digital agents, finding between 74.4 and 92.9% agreement with oracle evaluation metrics. Finally, we use these evaluators to improve the performance of existing agents via fine-tuning and inference-time guidance. Without any additional supervision, we improve state-of-the-art performance by 29% on the popular benchmark WebArena, and achieve around 75% relative improvement in device control settings.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
AWM uses 40.8 fewer steps on average than the AutoEval method on WebArena.
“Further compared to the Autoeval (Pan et al., 2024) method, which necessitates additional evaluation and refinement steps to solve tasks correctly, our AWM approach uses 40 . 8 fewer steps on average.”
From Agent Workflow Memory (2024) · cites this paper as Pan et al., 2024
Section 4.2 (p. 8) describes the cited method using an evaluator and up to three rounds of reflection and retry on WebArena, supporting the description of additional evaluation and refinement. Our copy does not report a comparison with AWM or the figure of 40.8 fewer steps; that numerical comparison is not supported by this source. (LLM, conf=0.87)
Experimental — accuracy not yet measuredPrevious efforts in web agents often focus narrowly on immediate, execution-level errors and struggle with more pervasive navigation challenges.
“Previous efforts in this domain often focus narrowly on immediate, execution-level errors and struggle with more pervasive navigation challenges (Shinn et al., 2023; Pan et al., 2024; Wang et al., 2024a).”
From R2D2: Remembering, Replaying and Dynamic Decision Making with a Reflective Agentic Memory (2025) · cites this paper as Pan et al., 2024
Related Work (p. 3) says that next-action matching does not capture the consequences of an agent’s sequential decisions, and the Introduction (p. 1) says even state-of-the-art agents make mistakes on simple tasks. These passages support limitations in evaluating and performing navigation tasks, but do not establish that previous efforts generally focus narrowly on immediate execution-level errors. (LLM, conf=0.83)
Experimental — accuracy not yet measuredThe LM-based evaluation model of Pan et al. (2024) outputs a binary label that judges whether an agent trajectory successfully solves a task.
“We adopt the LM-based evaluation model of Pan et al. (2024) to output a binary label, L eval ( e t ) ∈ { 0 , 1 } , that judges if e t successfully solves q t by prompting a neural model.”
From Agent Workflow Memory (2024) · cites this paper as Pan et al., 2024
On WebArena, SkillWeaver achieves higher average success rates and better or comparable performance across all domains compared to AutoEval, except in the Shopping environment.
“When compared to AutoEval (Pan et al., 2024), which leverages an LLM-based reward model to guide inference-time exploration, SKILLWEAVER achieves higher average success rates and exhibits better or comparable performance across all domains, with the exception of the Shopping environment.”
From SkillWeaver: Web Agents can Self-Improve by Discovering and Honing Skills (2025) · cites this paper as Pan et al., 2024
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
The publisher hasn’t deposited this paper’s reference list.
Research works printed but not linked: 43
- Marwa Abdulhai, Isadora White, Charles Burton Snell, Charles Sun, Joey Hong, Yuexiang Zhai, Kelvin Xu, and Sergey Levine. LMRL Gym: Benchmarks for multi-turn reinforcement learning with language models. ArXiv , abs/2311.18232, 2023. URL https://api.semanticscholar.org/CorpusID:265506611 . · arXiv
- James Allen, Nathanael Chambers, George Ferguson, Lucian Galescu, Hyuckchul Jung, Mary Swift, and William Taysom. PLOW: a collaborative task learning agent. In AAAI , 2007.
- Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-VL: A frontier large vision-language model with versatile abilities. arXiv , abs/2308.12966, 2023. URL https://api.semanticscholar.org/CorpusID: 263875678 . · arXiv
- Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Carol Chen, Catherine Olsson, Christopher Olah, Danny Hernandez, Dawn Drain, Deep Ganguli, Dustin Li, Eli Tran-Johnson, Ethan Perez, Jamie Kerr, Jared Mueller, Jeffrey Ladish, Joshua Landau, Kamal Ndousse, Kamile Lukosuite, Liane Lovitt, Michael Sellitto, Nelson Elhage, Nicholas Schiefer, Noemi Mercado, Nova DasSarma, Robert Lasenby, Robin Larson, Sam Ringer, Scott Johnston, Shauna Kravec, Sheer El Showk, Stanislav Fort, Tamera Lanham, Timothy Telleen-Lawton, Tom Conerly, Tom Henighan, Tristan Hume, Samuel R. Bowman, Zac Hatfield-Dodds, Ben Mann, Dario Amodei, Nicholas Joseph, Sam McCandlish, Tom Brown, and Jared Kaplan. Constitutional AI: Harmlessness from AI feedback. arXiv , abs/2308.12966, 2022. · arXiv
- S.R.K. Branavan, Harr Chen, Luke Zettlemoyer, and Regina Barzilay. Reinforcement learning for mapping instructions to actions. In ACL-AFNLP , 2009. URL https://aclanthology. org/P09-1010 .
- S.R.K. Branavan, Luke Zettlemoyer, and Regina Barzilay. Reading between the lines: Learning to map high-level instructions to commands. In ACL , 2010. URL https:// aclanthology.org/P10-1129 .
- Andrea Burns, Deniz Arsan, Sanjna Agrawal, Ranjitha Kumar, Kate Saenko, and Bryan A. Plummer. A dataset for interactive vision-language navigation with unknown command feasibility. In ECCV , 2022.
- Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Michael Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. In NeurIPS , 2021. URL https://openreview.net/forum? id=a7APmM4B9d .
- Xinyue Chen, Zijian Zhou, Zheng Wang, Che Wang, Yanqiu Wu, and Keith Ross. BAIL: Best-action imitation learning for batch deep reinforcement learning. In NeurIPS , 2020. URL https://proceedings.neurips.cc/paper files/paper/2020/file/ d55cbf210f175f4a37916eafe6c04f0d-Paper.pdf .
- Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Samuel Stevens, Boshi Wang, Huan Sun, and Yu Su. Mind2Web: Towards a generalist agent for the web. In NeurIPS Datasets and Benchmarks Track , 2023. URL https://openreview.net/forum?id=kiYqbO3wqw .
- Yuqing Du, Ksenia Konyushkova, Misha Denil, Akhil Raju, Jessica Landon, Felix Hill, Nando de Freitas, and Serkan Cabi. Vision-language models as success detectors. arXiv , 2023.
- Scott Emmons, Benjamin Eysenbach, Ilya Kostrikov, and Sergey Levine. RvS: What is essential for offline RL via supervised learning? In ICLR , 2022. URL https://openreview. net/forum?id=S874XAIpkR-.
- Jiaxian Guo, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Boyang Li, Dacheng Tao, and Steven Hoi. From images to textual prompts: Zero-shot visual question answering with frozen large language models. In CVPR , 2023.
- Izzeddin Gur, Hiroki Furuta, Austin V Huang, Mustafa Safdari, Yutaka Matsuo, Douglas Eck, and Aleksandra Faust. A real-world webagent with planning, long context understanding, and program synthesis. In ICLR , 2024. URL https://openreview.net/forum? id=9JQtrumvg8 .
- Hongliang He, Wenlin Yao, Kaixin Ma, Wenhao Yu, Yong Dai, Hongming Zhang, Zhenzhong Lan, and Dong Yu. WebVoyager: Building an end-to-end web agent with large multimodal models. ArXiv , abs/2401.13919, 2024. URL https://api.semanticscholar.org/CorpusID: 267211622 . · arXiv
- Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. NeurIPS , 30, 2017.
- Wenyi Hong, Weihan Wang, Qingsong Lv, Jiazheng Xu, Wenmeng Yu, Junhui Ji, Yan Wang, Zihan Wang, Yuxiao Dong, Ming Ding, et al. Cogagent: A visual language model for gui agents. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pp. 14281-14290, 2024.
- Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In ICLR , 2022. URL https://openreview.net/forum?id=nZeVKeeFYf9 .
- Peter C. Humphreys, David Raposo, Tobias Pohlen, Gregory Thornton, Rachita Chhaparia, Alistair Muldal, Josh Abramson, Petko Georgiev, Alex Goldin, Adam Santoro, and Timothy P. Lillicrap. A data-driven approach for learning to control computers. In ICML , 2022. URL https://api.semanticscholar.org/CorpusID:246867455 .
- Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de Las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, L´ elio Renard Lavaud, Lucile Saulnier, Marie-Anne Lachaux, Pierre Stock, Sandeep Subramanian, Sophia Yang, Szymon Antoniak, Teven Le Scao, Th´ eophile Gervet, Thibaut Lavril, Thomas Wang, Timoth´ ee Lacroix, and William El Sayed. Mixtral of experts. ArXiv , abs/2401.04088, 2024. URL https://api.semanticscholar.org/CorpusID:266844877 . · arXiv
- Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. CoRR , abs/1412.6980, 2014. URL https://api.semanticscholar.org/CorpusID:6628106 . · arXiv
- Jing Yu Koh, Robert Lo, Lawrence Jang, Vikram Duvvur, Ming Chong Lim, Po-Yu Huang, Graham Neubig, Shuyan Zhou, Ruslan Salakhutdinov, and Daniel Fried. Visualwebarena: Evaluating multimodal agents on realistic visual web tasks. ACL , 2024.
- Yang Li, Jiacong He, Xin Zhou, Yuan Zhang, and Jason Baldridge. Mapping natural language instructions to mobile UI action sequences. In ACL , 2020. URL https://aclanthology. org/2020.acl-main.729 .
- Evan Zheran Liu, Kelvin Guu, Panupong Pasupat, and Percy Liang. Reinforcement learning on web interfaces using workflow-guided exploration. In ICLR , 2018. URL https: //openreview.net/forum?id=ryTp3f-0-.
- Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke E. Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Francis Christiano, Jan Leike, and Ryan J. Lowe. Training language models to follow instructions with human feedback. ArXiv , abs/2203.02155, 2022. URL https: //api.semanticscholar.org/CorpusID:246426909 . · arXiv
- Christopher Rawles, Alice Li, Daniel Rodriguez, Oriana Riva, and Timothy P Lillicrap. AndroidInTheWild: A large-scale dataset for android device control. In NeurIPS Datasets and Benchmarks Track , 2023. URL https://openreview.net/forum?id=j4b3l5kOil .
- Tianlin Shi, Andrej Karpathy, Linxi Fan, Jonathan Hernandez, and Percy Liang. World of Bits: An open-domain platform for web-based agents. In ICML , 2017. URL https: //proceedings.mlr.press/v70/shi17a.html .
- Noah Shinn, Federico Cassano, Beck Labash, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: language agents with verbal reinforcement learning. In NeurIPS , 2023. URL https://api.semanticscholar.org/CorpusID:258833055 .
- Daniel Toyama, Philippe Hamel, Anita Gergely, Gheorghe Comanici, Amelia Glaese, Zafarali Ahmed, Tyler Jackson, Shibl Mourad, and Doina Precup. AndroidEnv: A reinforcement learning platform for android. arXiv , abs/2105.13231, 2021. URL https: //api.semanticscholar.org/CorpusID:235212182 . · arXiv
- Junyang Wang, Haiyang Xu, Jiabo Ye, Ming Yan, Weizhou Shen, Ji Zhang, Fei Huang, and Jitao Sang. Mobile-Agent: Autonomous multi-modal mobile device agent with visual perception. arXiv , abs/2401.16158, 2024. · arXiv
- Ziyue Wang, Chi Chen, Peng Li, and Yang Liu. Filling the image information gap for VQA: Prompting large language models to proactively ask questions. 2023b. URL https://aclanthology.org/2023.findings-emnlp.189 .
- Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. NeurIPS , 2022.
- Zhiyong Wu, Chengcheng Han, Zichen Ding, Zhenmin Weng, Zhoumianze Liu, Shunyu Yao, Tao Yu, and Lingpeng Kong. OS-Copilot: Towards generalist computer agents with self-improvement. arXiv , abs/2402.07456, 2024. · arXiv
- Nancy Xu, Sam Masling, Michael Du, Giovanni Campagna, Larry Heck, James Landay, and Monica Lam. Grounding open-domain instructions to automate web support tasks. In NAACL-HLT , 2021. URL https://aclanthology.org/2021.naacl-main.80 .
- An Yan, Zhengyuan Yang, Wanrong Zhu, Kevin Qinghong Lin, Linjie Li, Jianfeng Wang, Jianwei Yang, Yiwu Zhong, Julian McAuley, Jianfeng Gao, Zicheng Liu, and Lijuan Wang. GPT-4V in Wonderland: Large multimodal models for zero-shot smartphone GUI navigation. arXiv , abs/2311.07562, 2023. URL https://api.semanticscholar.org/ CorpusID:265149992 . · arXiv
- Shunyu Yao, Howard Chen, John Yang, and Karthik Narasimhan. WebShop: Towards scalable real-world web interaction with grounded language agents. In NeurIPS , 2022. URL https://proceedings.neurips.cc/paper files/paper/2022/file/ 82ad13ec01f9fe44c01cb91814fd7b8c-Paper-Conference.pdf .
- Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao, and Karthik R. Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. In NeurIPS , 2023. URL https://openreview.net/forum?id=5Xc1ecxO1h .
- Haoxuan You, Rui Sun, Zhecan Wang, Long Chen, Gengyu Wang, Hammad Ayyubi, Kai-Wei Chang, and Shih-Fu Chang. IdealGPT: Iteratively decomposing vision and language reasoning via large language models. In Findings of EMNLP , 2023. URL https://aclanthology.org/2023.findings-emnlp.755 .
- Chaoyun Zhang, Liqun Li, Shilin He, Xu Zhang, Bo Qiao, Si Qin, Minghua Ma, Yu Kang, Qingwei Lin, Saravan Rajmohan, et al. UFO: A UI-focused agent for Windows OS interaction. arXiv , abs/2402.07939, 2024. · arXiv
- Chi Zhang, Zhao Yang, Jiaxuan Liu, Yucheng Han, Xin Chen, Zebiao Huang, Bin Fu, and Gang Yu. AppAgent: Multimodal agents as smartphone users. arXiv , abs/2312.13771, 2023. · arXiv
- Zhuosheng Zhang and Aston Zhang. You only look at screens: Multimodal chain-of-action agents. arXiv , abs/2309.11436, 2023. URL https://api.semanticscholar.org/CorpusID: 262053313 . · arXiv
- Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. NeurIPS , 36, 2024.
- Shuyan Zhou, Frank F. Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, Uri Alon, and Graham Neubig. WebArena: A realistic web environment for building autonomous agents. In ICLR , 2024. URL https://openreview.net/forum?id=oKn9c6ytLx .
Cited by
Sorted by citations
OpenHands: An Open Platform for AI Software Developers as Generalist Agents
Wang, X., Li, B., Song, Y., Xu, F. F., Tang, X., Zhuge, M., Pan, J., Song, Y., Li, B., Singh, J., et al. (2024) · arXiv · doi:10.48550/arxiv.2407.16741 · Cited by 24 · see citing works
Self-citationYager, K. G. (2024) · Digital Discovery · doi:10.1039/d4dd00178h · Cited by 12 · see citing works
Wang, Z. Z., Mao, J., Fried, D., & Neubig, G. (2024) · arXiv · doi:10.48550/arxiv.2409.07429 · Cited by 5 · see citing works
“Further compared to the Autoeval (Pan et al., 2024) method, which necessitates additional evaluation and refinement steps to solve tasks correctly, our AWM approach uses 40 .”
MethodInfluentialScribeAgent: Towards Specialized Web Agents Using Production-Scale Workflow DataPreprint
Shen, J.-H., Jain, A., Xiao, Z., Amlekar, I., Hadji, M., Podolny, A., & Talwalkar, A. (2024) · Qeios · doi:10.32388/8vog0o · Cited by 13 · see citing works
How Memory Management Impacts LLM Agents: An Empirical Study of Experience-Following Behavior
Xiong, Z., Lin, Y., Xie, W., He, P., Liu, Z., Tang, J., Lakkaraju, H., & Xiang, Z. (2026) · Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2026.acl-long.27 · Cited by 4 · see citing works
R2D2: Remembering, Replaying and Dynamic Decision Making with a Reflective Agentic Memory
Huang, T., Basu, K., Abdelaziz, I., Kapanipathi, P., May, J., & Chen, M. (2025) · Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2025.acl-long.1464 · Cited by 4 · see citing works
“Previous efforts in this domain often focus narrowly on immediate, execution-level errors and struggle with more pervasive navigation challenges (Shinn et al., 2023; Pan et al., 2024; Wang et al., 2024a).”
InfluentialMind2Web 2: Evaluating Agentic Search with Agent-as-a-Judge
Gou, B., Huang, Z., Ning, Y., Gu, Y., Lin, M., Qi, W., Kopanev, A., Yu, B., Gutiérrez, B. J., Shu, Y., et al. (2025) · arXiv · doi:10.48550/arxiv.2506.21506 · Cited by 3 · see citing works
ReasoningBank: Scaling Agent Self-Evolving with Reasoning Memory
Ouyang, S., Yan, J., Hsu, I.-H., Chen, Y., Jiang, K., Wang, Z., Han, R., Le, L. T., Daruki, S., Tang, X., et al. (2025) · arXiv · doi:10.48550/arxiv.2509.25140
“You can extract at most 3 memory items from the trajectory.”
WebAgent-R1: Training Web Agents via End-to-End Multi-Turn Reinforcement Learning
Wei, Z., Yao, W., Liu, Y., Zhang, W., Lu, Q., Qiu, L., Yu, C., Xu, P., Zhang, C., Yin, B., et al. (2025) · arXiv · doi:10.48550/arxiv.2505.16421
“However, most of this line of research has heavily relied on offline or iterative off-policy RL solutions (Peng et al., 2019; Pan et al., 2024; Qi et al., 2025), which break the end-to-end interaction between the web agent and environment, and introduce additional complexities such as trajectory…”
SkillWeaver: Web Agents can Self-Improve by Discovering and Honing Skills
Zheng, B., Fatemi, M. Y., Jin, X., Wang, Z. Z., Gandhi, A., Song, Y., Gu, Y., Srinivasa, J., Liu, G., Neubig, G., et al. (2025) · arXiv · doi:10.48550/arxiv.2504.07079
Digital Agents and Generative Artificial Intelligence in Support of Logistics 5.0
Nicoletti, B., & Appolloni, A. (2025) · Lecture notes in information systems and organisation · doi:10.1007/978-3-032-00118-4_7
- Authors
- Pan, J., Zhang, Y., Tomlin, N., Zhou, Y., Levine, S., & Suhr, A.
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2024
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2404.06474
- OpenAlex
- W4394776438
- Cited by
- 3
- References
- 0
- Citations by year
- 2025: 1 · 2024: 2
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › AI-based Problem Solving and Planning
- Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
- Physical Sciences › Computer Science › Information Systems › Web Data Mining and Analysis
- Keywords
- digital agents, inference cost, fine-tuning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
