Agent-E: From Autonomous Web Navigation to Foundational Design Principles in Agentic Systems
Open accessAbstract
AI Agents are changing the way work gets done, both in consumer and enterprise domains. However, the design patterns and architectures to build highly capable agents or multi-agent systems are still developing, and the understanding of the implication of various design choices and algorithms is still evolving. In this paper, we present our work on building a novel web agent, Agent-E \footnote{Our code is available at \url{https://github.com/EmergenceAI/Agent-E}}. Agent-E introduces numerous architectural improvements over prior state-of-the-art web agents such as hierarchical architecture, flexible DOM distillation and denoising method, and the concept of \textit{change observation} to guide the agent towards more accurate performance. We first present the results of an evaluation of Agent-E on WebVoyager benchmark dataset and show that Agent-E beats other SOTA text and multi-modal web agents on this benchmark in most categories by 10-30\%. We then synthesize our learnings from the development of Agent-E into general design principles for developing agentic systems. These include the use of domain-specific primitive skills, the importance of distillation and de-noising of environmental observations, the advantages of a hierarchical architecture, and the role of agentic self-improvement to enhance agent efficiency and efficacy as the agent gathers experience.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Agent-E is a commercial web agent that operates as a Chrome extension, overlaying a chat window on the browser webpage for user interaction.
“Agent-E [24], a commercial web agent, operates as a Chrome extension, overlaying a chat window on the browser webpage for user interaction.”
From Investigating the Impact of Dark Patterns on LLM-Based Web Agents (2026) · cites this paper as [24]
The Introduction identifies Agent-E as a web agent, and Section 2 describes browser control using Playwright. The copy does not describe Agent-E as commercial or as a Chrome extension with a chat window overlaid on the webpage; Section 2’s “nested chat” refers to communication between agents, not a user interface. (LLM, conf=0.88)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
No references could be resolved yet.
Research works printed but not linked: 25
- Bacon, P.-L., Harb, J., and Precup, D. (2017). The option-critic architecture. AAAI Conference on Artificial Intelligence.
- Bai, H., Zhou, Y., Cemri, M., Pan, J., Suhr, A., Levine, S., and Kumar, A. (2024). Digirl: Training in-the-wild device-control agents with autonomous reinforcement learning. arXiv preprint arXiv:2406.11896. · arXiv
- Deng, X., Gu, Y., Zheng, B., Chen, S., Stevens, S., Wang, B., Sun, H., and Su, Y. (2024). Mind2web: Towards a generalist agent for the web. Advances in Neural Information Processing Systems, 36.
- He et al., 2024] He, H., Yao, W., Ma, K., Yu, W., Dai, Y., Zhang, H., Lan, Z., and Yu, D. (2024). Webvoyager: Building an end-to-end web agent with large multimodal models. arXiv preprint arXiv:2401.13919. · arXiv
- Hosseini, S., Awadallah, A. H., and Su, Y. (2021). Compositional generalization for natural language interfaces to web apis. arXiv preprint arXiv:2112.05209. · arXiv
- Irpan, A., Herzog, A., Toshev, A. T., Zeng, A., Brohan, A., Ichter, B. A., David, B., Parada, C., Finn, C., Tan, C., Reyes, D., Kalashnikov, D., Jang, E. V., Xia, F., Rettinghouse, J. L., Hsu, J. C., Quiambao, J. L., Ibarz, J., Rao, K., Hausman, K., Gopalakrishnan, K., Lee, K.-H., Jeffrey, K. A., Luu, L., Yan, M., Ahn, M. S., Sievers, N., Joshi, N. J., Brown, N., Cortes, O. E. E., Xu, P., Sampedro, P. P., Sermanet, P., Ruano, R. J., Julian, R. C., Jesmonth, S. A., Levine, S., Xu, S., Xiao, T., Vanhoucke, V. O., Lu, Y., Chebotar, Y., and Kuang, Y. (2022). Do as i can, not as i say: Grounding language in robotic affordances. arXiv preprint arXiv:2204.01691. · arXiv
- Kambhampati, S., Valmeekam, K., Guan, L., Verma, M., Stechly, K., Bhambri, S., Saldyt, L., and Murthy, A. (2024). Llms can't plan, but can help planning in llm- modulo frameworks. arXiv preprint arXiv:2402.01817. · arXiv
- Kapoor, S., Stroebl, B., Siegel, Z. S., Nadgir, N., and Narayanan, A. (2024). Ai agents that matter. arXiv preprint arXiv:2407.01502. · arXiv
- Lutz, M., Bohra, A., Saroyan, M., Harutyunyan, A., and Campagna, G. (2024). Wilbur: Adaptive in-context learning for robust and accurate web agents. arXiv preprint arXiv:2404.05902. · arXiv
- Marthi, B., Russell, S., and Wolfe, J. (2007). Angelic semantics for high-level actions. International Conference on Automated Planning and Scheduling.
- Nakano, R., Hilton, J., Balaji, S., Wu, J., Ouyang, L., Kim, C., Hesse, C., Jain, S., Kosaraju, V., Saunders, W., Jiang, X., Cobbe, K., Eloundou, T., Krueger, G., Button, K., Knight, M., Chess, B., and Schulman, J. (2022). Webgpt: Browser-assisted question-answering with human feedback. arXiv preprint arXiv:2112.09332. · arXiv
- Nau, D., Cao, Y., Lotem, A., and Muñoz-Avila, H. (1991). Shop: Simple hierarchical ordered planner. International Joint Conference on Artificial Intelligence.
- Patil, S. G., Zhang, T., Wang, X., and Gonzalez, J. E. (2023). Gorilla: Large language model connected with massive apis. arXiv preprint arXiv:2305.15334. · arXiv
- Qin, Y., Liang, S., Ye, Y., Zhu, K., Yan, L., Lu, Y., Lin, Y., Cong, X., Tang, X., Qian, B., Zhao, S., Hong, L., Tian, R., Xie, R., Zhou, J., Gerstein, M., Li, D., Liu, Z., and Sun, M. (2024). Toolllm: Facilitating large language models to master 16000+ real-world apis. International Conference on Learning Representations.
- Russell, S. J. and Norvig, P. (2009). Artificial Intelligence: a modern approach. Pearson, 3 edition.
- Shinn, N., Cassano, F., Gopinath, A., Narasimhan, K., and Yao, S. (2024). Reflex- ion: Language agents with verbal reinforcement learning. Advances in Neural Information Processing Systems, 36.
- Sutton, R., Precup, D., and Singh, S. (1999). Between mdps and semi-mdps: A framework for temporal abstraction in reinforcement learning. Artificial Intelligence Journal.
- Tate, A. (1977). Generating project networks. International Joint Conference on Artificial Intelligence.
- Wang, Z., Cai, S., Chen, G., Liu, A., Ma, X., and Liang, Y. (2022). Describe, explain, plan and select: Interactive planning with large language models enables open-world multi- task agents. Advances in Neural Information Processing Systems, 37.
- Wen, H., Li, Y., Liu, G., Zhao, S., Yu, T., Li, T. J.-J., Jiang, S., Liu, Y., Zhang, Y., and Liu, Y. (2024). Autodroid: Llm-powered task automation in android. In Proceedings of the 30th Annual International Conference on Mobile Computing and Networking, pages 543-557.
- Wornow, M., Narayan, A., Opsahl-Ong, K., McIntyre, Q., Shah, N. H., and Re, C. (2024). Automating the enterprise with foundation models. arXiv preprint arXiv:2405.03710. · arXiv
- Wu et al., 2023b] Wu, Q., Bansal, G., Zhang, J., Wu, Y., Zhang, S., Zhu, E., Li, B., Jiang, L., Zhang, X., and Wang, C. (2023b). Autogen: Enabling next-gen llm applications via multi-agent conversation framework. arXiv preprint arXiv:2308.08155. · arXiv
- Yao, S., Chen, H., Yang, J., and Narasimhan, K. (2022). Webshop: Towards scal- able real-world web interaction with grounded language agents. Advances in Neural Information Processing Systems, 35:20744-20757.
- Zheng, B., Gou, B., Kil, J., Sun, H., and Su, Y. (2024). Gpt-4v (ision) is a generalist web agent, if grounded. arXiv preprint arXiv:2401.01614. · arXiv
- Zhou, S., Xu, F. F., Zhu, H., Zhou, X., Lo, R., Sridhar, A., Cheng, X., Bisk, Y., Fried, D., Alon, U., et al. (2023). Webarena: A realistic web environment for building autonomous agents. arXiv preprint arXiv:2307.13854. · arXiv
Cited by
Sorted by citations
WebWalker: Benchmarking LLMs in Web Traversal
Wu, J., Yin, W., Jiang, Y., Wang, Z., Xi, Z., Fang, R., Zhang, L., He, Y., Zhou, D., Xie, P., et al. (2025) · Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2025.acl-long.508 · Cited by 18 · see citing works
DAWN: Designing Distributed Agents in a Worldwide Network
Aminiranjbar, Z., Tang, J., Wang, Q., Pant, S., & Viswanathan, M. (2024) · arXiv · doi:10.48550/arxiv.2410.22339 · Cited by 8 · see citing works
Investigating the Impact of Dark Patterns on LLM-Based Web Agents
Ersoy, D., Lee, B., Shreekumar, A., Arunasalam, A., Ibrahim, M., Bianchi, A., & Berkay Celik, Z. (2026) · 2026 IEEE Symposium on Security and Privacy (SP) · doi:10.1109/sp63933.2026.00042 · Cited by 1 · see citing works
WebAgent-R1: Training Web Agents via End-to-End Multi-Turn Reinforcement Learning
Wei, Z., Yao, W., Liu, Y., Zhang, W., Lu, Q., Qiu, L., Yu, C., Xu, P., Zhang, C., Yin, B., et al. (2025) · arXiv · doi:10.48550/arxiv.2505.16421
“…to a variety of real-world interactive tasks, including web navigation (Nakano et al., 2021; Yao et al., 2022; Ma et al., 2023; Gur et al., 2024; Abuelsaad et al., 2024; Lutz et al., 2024; Pa-tel et al., 2024; Putta et al., 2024), general computer use (Li et al., 2020; Deng et al., 2023; Yang…”
ST-WebAgentBench: A Benchmark for Evaluating Safety and Trustworthiness in Web Agents
Levy, I., Wiesel, B., Marreed, S., Oved, A., Yaeli, A., Mashkif, N., & Shlomov, S. (2024) · arXiv · doi:10.48550/arxiv.2410.06703 · Cited by 1 · see citing works
- Authors
- Abuelsaad, T., Akkil, D., Dey, P., Jagmohan, A., Vempaty, A., & Kokku, R.
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2024
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2407.13032
- OpenAlex
- W4402346179
- Cited by
- 31
- References
- 0
- Citations by year
- 2026: 15 · 2025: 16
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Information Systems › Web Data Mining and Analysis
- Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
- Physical Sciences › Computer Science › Artificial Intelligence › AI-based Problem Solving and Planning
- Keywords
- web agents, hierarchy, multi-agent systems, human development
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
