CoChat
Sign Up
PreprintCited by 15

The DOI matches its DataCite record.

WebVoyager: Building an End-to-End Web Agent with Large Multimodal Models

He, Hongliang, Yao, Wenlin, Ma, Kaixin, Yu, Wenhao, Dai, Yong, Zhang, Hongming, Lan, Zhenzhong, Yu, Dong·2024·arXiv
DOI 10.48550/arxiv.2401.13919arXiv 2401.13919
Open this paper in CoChat — free account

Abstract

The rapid advancement of large language models (LLMs) has led to a new era marked by the development of autonomous applications in real-world scenarios, which drives innovation in creating advanced web agents. Existing web agents typically only handle one input modality and are evaluated only in simplified web simulators or static web snapshots, greatly limiting their applicability in real-world scenarios. To bridge this gap, we introduce WebVoyager, an innovative Large Multimodal Model (LMM) powered web agent that can complete user instructions end-to-end by interacting with real-world websites. Moreover, we establish a new benchmark by compiling real-world tasks from 15 popular websites and introduce an automatic evaluation protocol leveraging multimodal understanding abilities of GPT-4V to evaluate open-ended web agents. We show that WebVoyager achieves a 59.1% task success rate on our benchmark, significantly surpassing the performance of both GPT-4 (All Tools) and the WebVoyager (text-only) setups, underscoring the exceptional capability of WebVoyager. The proposed automatic evaluation metric achieves 85.3% agreement with human judgment, indicating its effectiveness in providing reliable and accurate assessments of web agents.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2401.13919

Cites

Statistics

Figures

Figure 1 The overall workflow of WebVoyager. WebVoyager takes web tasks assigned by a human and automatically browses the web online. At each step, WebVoyager selects actions based on screenshots and text (the 'type' of the web element and its contents). Once the task is completed, the answers will be returned to the user. For example, for a user query: "Find the cost of a 2-year protection for PS4 on Amazon.", the agent interacts with Amazon online, locates the PS4, identifies the 2-year protection price, and returns "$30.99" to the user.
Figure 1 The overall workflow of WebVoyager. WebVoyager takes web tasks assigned by a human and automatically browses the web online. At each step, WebVoyager selects actions based on screenshots and text (the 'type' of the web element and its contents). Once the task is completed, the answers will be returned to the user. For example, for a user query: "Find the cost of a 2-year protection for PS4 on Amazon.", the agent interacts with Amazon online, locates the PS4, identifies the 2-year protection price, and returns "$30.99" to the user.CC-BY · From the PDF
Figure 2 Examples of webpage screenshots provided to the agent. We add borders to most of the interactive elements on the web pages and label them with numerical tags in the top left corner.
Figure 2 Examples of webpage screenshots provided to the agent. We add borders to most of the interactive elements on the web pages and label them with numerical tags in the top left corner.CC-BY · From the PDF
Figure 3 Data creation process using self-instruct.
Figure 3 Data creation process using self-instruct.CC-BY · From the PDF
Figure 4 Screenshots of a complete trajectory of online web browsing. Given the task: 'Search Apple for the accessory Smart Folio for iPad and check the closest pickup availability next to zip code 90038.' The agent interacts with the Apple website and obtains the answer: 'Apple Tower Theatre.'
Figure 4 Screenshots of a complete trajectory of online web browsing. Given the task: 'Search Apple for the accessory Smart Folio for iPad and check the closest pickup availability next to zip code 90038.' The agent interacts with the Apple website and obtains the answer: 'Apple Tower Theatre.'CC-BY · From the PDF
Figure 5 Success Rate for GAIA Level 1 and Level 2.
Figure 5 Success Rate for GAIA Level 1 and Level 2.CC-BY · From the PDF
Figure 6 Factors related to task success rate. We show the average number of elements per page and the average trajectory length for each website type. The darker colors indicate a higher task success rate.
Figure 6 Factors related to task success rate. We show the average number of elements per page and the average trajectory length for each website type. The darker colors indicate a higher task success rate.CC-BY · From the PDF
Figure 7 System Prompt for WebVoyager. We instruct agents to perform web navigation, along with specific browsing actions and action formats. To enhance efficiency and accuracy, we can incorporate additional general into the prompts. These guidelines should be generic and not about a specific website to ensure generalizability.
Figure 7 System Prompt for WebVoyager. We instruct agents to perform web navigation, along with specific browsing actions and action formats. To enhance efficiency and accuracy, we can incorporate additional general into the prompts. These guidelines should be generic and not about a specific website to ensure generalizability.CC-BY · From the PDF
Figure 8 System Prompt for Auto Evaluation by GPT-4V.
Figure 8 System Prompt for Auto Evaluation by GPT-4V.CC-BY · From the PDF
Image 9
CC-BY · From the PDF
Image 10
CC-BY · From the PDF
Step 7: ANSWER
Step 7: ANSWERCC-BY · From the PDF
Step 1: Click [10]
Step 1: Click [10]CC-BY · From the PDF
Step 1: Type [15]; Jakarta
Step 1: Type [15]; JakartaCC-BY · From the PDF
Image 14
CC-BY · From the PDF
Image 15
CC-BY · From the PDF
Image 16
CC-BY · From the PDF
Image 17
CC-BY · From the PDF
Image 18
CC-BY · From the PDF
Image 19
CC-BY · From the PDF
Image 20
CC-BY · From the PDF
Image 21
CC-BY · From the PDF
Image 22
CC-BY · From the PDF
Image 23
CC-BY · From the PDF
Image 24
CC-BY · From the PDF
Image 25
CC-BY · From the PDF
Step 4: Type [2]; Xi Yang
Step 4: Type [2]; Xi YangCC-BY · From the PDF
Step 4: Scroll down
Step 4: Scroll downCC-BY · From the PDF
Image 28
CC-BY · From the PDF
Figure 17 Screenshots of a complete trajectory of browsing GitHub. Given the task: 'Search for an open-source project related to 'climate change data visualization' on GitHub and report the project with the most stars.' The agent interacts with the GitHub website and obtains the answer: 'resource-watch/resource-watch with 63 stars'
Figure 17 Screenshots of a complete trajectory of browsing GitHub. Given the task: 'Search for an open-source project related to 'climate change data visualization' on GitHub and report the project with the most stars.' The agent interacts with the GitHub website and obtains the answer: 'resource-watch/resource-watch with 63 stars'CC-BY · From the PDF
Figure 19 Screenshots of a complete trajectory of browsing Google Flights. Given the task: 'Search a one-way flight from Dublin To Athens Greece for 1 Adult that leaves on December 30 and analyse the price graph for the next 2 months.' The agent interacts with the Google Flights website and obtains the answer: 'The price graph has been analyzed, and it shows the trends for flight prices from Dublin to Athens over the next two months.'
Figure 19 Screenshots of a complete trajectory of browsing Google Flights. Given the task: 'Search a one-way flight from Dublin To Athens Greece for 1 Adult that leaves on December 30 and analyse the price graph for the next 2 months.' The agent interacts with the Google Flights website and obtains the answer: 'The price graph has been analyzed, and it shows the trends for flight prices from Dublin to Athens over the next two months.'CC-BY · From the PDF
Image 31
CC-BY · From the PDF
Figure 20 Screenshots of a complete trajectory of browsing Google Search. Given the task: 'Show me a list of comedy movies, sorted by user ratings. Show me the Top 5 movies.' The agent interacts with the Google Search website and obtains the answer: 'The Top 5 comedy movies sorted by user ratings are:Life Is Beautiful; Back to the Future; The Intouchables; City Lights; Modern Times'
Figure 20 Screenshots of a complete trajectory of browsing Google Search. Given the task: 'Show me a list of comedy movies, sorted by user ratings. Show me the Top 5 movies.' The agent interacts with the Google Search website and obtains the answer: 'The Top 5 comedy movies sorted by user ratings are:Life Is Beautiful; Back to the Future; The Intouchables; City Lights; Modern Times'CC-BY · From the PDF
Step 1: Type [8]; x^5-20x^4
Step 1: Type [8]; x^5-20x^4CC-BY · From the PDF
Figure 23 Screenshots of a complete trajectory of browsing Google Flights in Chinese. Given the task: 'Find a flight from Hangzhou to Shenzhen on June 15, 2024, returning on June 20, departing at night, answer a flight that meets the requirements, and answer its specific departure time, airline and price.' The agent interacts with the Google Flights website and obtains the answer: 'Departure time: 17:35, airline: Shenzhen Airlines, price: HK$2,680'
Figure 23 Screenshots of a complete trajectory of browsing Google Flights in Chinese. Given the task: 'Find a flight from Hangzhou to Shenzhen on June 15, 2024, returning on June 20, departing at night, answer a flight that meets the requirements, and answer its specific departure time, airline and price.' The agent interacts with the Google Flights website and obtains the answer: 'Departure time: 17:35, airline: Shenzhen Airlines, price: HK$2,680'CC-BY · From the PDF
Image 35
CC-BY · From the PDF
Figure 24 Screenshots of a complete trajectory of browsing Cambridge Dictionary in Spanish. The description of task and answer are shown in Figure.
Figure 24 Screenshots of a complete trajectory of browsing Cambridge Dictionary in Spanish. The description of task and answer are shown in Figure.CC-BY · From the PDF
Image 37
CC-BY · From the PDF
Step 4: Click [13]
Step 4: Click [13]CC-BY · From the PDF
Image 39
CC-BY · From the PDF

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

No references could be resolved yet.

Research works printed but not linked: 41

  1. Armen Aghajanyan, Bernie Huang, Candace Ross, Vladimir Karpukhin, Hu Xu, Naman Goyal, Dmytro Okhonko, Mandar Joshi, Gargi Ghosh, Mike Lewis, et al. 2022. Cm3: A causal masked multi- modal model of the internet. arXiv preprint arXiv:2201.07520. · arXiv
  2. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901.
  3. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Ka- plan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374. · arXiv
  4. Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebas- tian Gehrmann, et al. 2023. Palm: Scaling language modeling with pathways. Journal of Machine Learn- ing Research, 24(240):1-113.
  5. Jacob Cohen. 1960. A coefficient of agreement for nominal scales. Educational and psychological mea- surement, 20(1):37-46.
  6. Yong Dai, Duyu Tang, Liangxin Liu, Minghuan Tan, Cong Zhou, Jingquan Wang, Zhangyin Feng, Fan Zhang, Xueyu Hu, and Shuming Shi. 2022. One model, multiple modalities: A sparsely activated ap- proach for text, sound, image, video and code. arXiv preprint arXiv:2205.06126. · arXiv
  7. Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Samuel Stevens, Boshi Wang, Huan Sun, and Yu Su. 2023. Mind2web: Towards a generalist agent for the web. arXiv preprint arXiv:2306.06070. · arXiv
  8. Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Zhiy- ong Wu, Baobao Chang, Xu Sun, Jingjing Xu, and Zhifang Sui. 2022. A survey for in-context learning. arXiv preprint arXiv:2301.00234. · arXiv
  9. Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. 2020. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929. · arXiv
  10. Joseph L Fleiss. 1971. Measuring nominal scale agree- ment among many raters. Psychological bulletin, 76(5):378.
  11. Hiroki Furuta, Ofir Nachum, Kuang-Huei Lee, Yu- taka Matsuo, Shixiang Shane Gu, and Izzeddin Gur. 2023. Multimodal web navigation with instruction-finetuned foundation models. arXiv preprint arXiv:2305.11854. · arXiv
  12. Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. 2017. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6904-6913.
  13. Izzeddin Gur, Hiroki Furuta, Austin Huang, Mustafa Safdari, Yutaka Matsuo, Douglas Eck, and Aleksan- dra Faust. 2023. A real-world webagent with plan- ning, long context understanding, and program syn- thesis. arXiv preprint arXiv:2307.12856. · arXiv
  14. Jack Hessel, Jena D Hwang, Jae Sung Park, Rowan Zellers, Chandra Bhagavatula, Anna Rohrbach, Kate Saenko, and Yejin Choi. 2022. The abduction of sherlock holmes: A dataset for visual abductive rea- soning. In European Conference on Computer Vision, pages 558-575. Springer.
  15. Kenton Lee, Mandar Joshi, Iulia Raluca Turc, Hexi- ang Hu, Fangyu Liu, Julian Martin Eisenschlos, Ur- vashi Khandelwal, Peter Shaw, Ming-Wei Chang, and Kristina Toutanova. 2023. Pix2struct: Screen- shot parsing as pretraining for visual language under- standing. In International Conference on Machine Learning, pages 18893-18912.
  16. PMLR.
  17. Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai-Wei Chang. 2019. Visualbert: A sim- ple and performant baseline for vision and language. arXiv preprint arXiv:1908.03557. · arXiv
  18. Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2024. Visual instruction tuning. Advances in neural information processing systems, 36.
  19. Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai- Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. 2022. Learn to explain: Multimodal reasoning via thought chains for science question answering. Advances in Neural Information Processing Systems, 35:2507-2521.
  20. Pan Lu, Baolin Peng, Hao Cheng, Michel Galley, Kai- Wei Chang, Ying Nian Wu, Song-Chun Zhu, and Jian- feng Gao. 2023. Chameleon: Plug-and-play compo- sitional reasoning with large language models. arXiv preprint arXiv:2304.09842. · arXiv
  21. Kaixin Ma, Hongming Zhang, Hongwei Wang, Xiao- man Pan, and Dong Yu. 2023. Laser: Llm agent with state-space exploration for web navigation. arXiv preprint arXiv:2309.08172. · arXiv
  22. Grégoire Mialon, Clémentine Fourrier, Craig Swift, Thomas Wolf, Yann LeCun, and Thomas Scialom. 2023. Gaia: a benchmark for general ai assistants. arXiv preprint arXiv:2311.12983. · arXiv
  23. Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, et al. 2021. Webgpt: Browser-assisted question- answering with human feedback. arXiv preprint arXiv:2112.09332. · arXiv
  24. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instruc- tions with human feedback. Advances in Neural Information Processing Systems, 35:27730-27744.
  25. Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, et al. 2023. Toolllm: Facilitating large language models to master 16000+ real-world apis. arXiv preprint arXiv:2307.16789. · arXiv
  26. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text trans- former. The Journal of Machine Learning Research, 21(1):5485-5551.
  27. Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. 2023. Toolformer: Language models can teach themselves to use tools. arXiv preprint arXiv:2302.04761. · arXiv
  28. Peter Shaw, Mandar Joshi, James Cohan, Jonathan Be- rant, Panupong Pasupat, Hexiang Hu, Urvashi Khan- delwal, Kenton Lee, and Kristina Toutanova. 2023. From pixels to ui actions: Learning to follow instruc- tions via graphical user interfaces. arXiv preprint arXiv:2306.00245. · arXiv
  29. Tianlin Shi, Andrej Karpathy, Linxi Fan, Jonathan Her- nandez, and Percy Liang. 2017. World of bits: An open-domain platform for web-based agents. In In- ternational Conference on Machine Learning, pages 3135-3144. PMLR.
  30. Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik R Narasimhan, and Shunyu Yao. 2023. Re- flexion: Language agents with verbal reinforcement learning. In Thirty-seventh Conference on Neural Information Processing Systems.
  31. Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. 2023. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805. · arXiv
  32. Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Al- isa Liu, Noah A Smith, Daniel Khashabi, and Han- naneh Hajishirzi. 2022. Self-instruct: Aligning lan- guage model with self generated instructions. arXiv preprint arXiv:2212.10560. · arXiv
  33. Zirui Wang, Jiahui Yu, Adams Wei Yu, Zihang Dai, Yu- lia Tsvetkov, and Yuan Cao. 2021. Simvlm: Simple visual language model pretraining with weak super- vision. arXiv preprint arXiv:2108.10904. · arXiv
  34. Jianwei Yang, Hao Zhang, Feng Li, Xueyan Zou, Chun- yuan Li, and Jianfeng Gao. 2023a. Set-of-mark prompting unleashes extraordinary visual grounding in gpt-4v. arXiv preprint arXiv:2310.11441. · arXiv
  35. Zhengyuan Yang, Linjie Li, Kevin Lin, Jianfeng Wang, Chung-Ching Lin, Zicheng Liu, and Lijuan Wang. 2023b. The dawn of lmms: Preliminary explorations with gpt-4v (ision). arXiv preprint arXiv:2309.17421, 9(1). · arXiv
  36. Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. 2022b. React: Synergizing reasoning and acting in language models. arXiv preprint arXiv:2210.03629. · arXiv
  37. Da Yin, Faeze Brahman, Abhilasha Ravichander, Khy- athi Chandu, Kai-Wei Chang, Yejin Choi, and Bill Yuchen Lin. 2023. Lumos: Learning agents with unified data, modular design, and open-source llms. arXiv preprint arXiv:2311.05657. · arXiv
  38. Rowan Zellers, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. From recognition to cognition: Vi- sual commonsense reasoning. In Proceedings of the IEEE/CVF conference on computer vision and pat- tern recognition, pages 6720-6731.
  39. Boyuan Zheng, Boyu Gou, Jihyung Kil, Huan Sun, and Yu Su. 2024. Gpt-4v (ision) is a generalist web agent, if grounded. arXiv preprint arXiv:2401.01614. · arXiv
  40. Shuyan Zhou, Frank F Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Yonatan Bisk, Daniel Fried, Uri Alon, et al. 2023. Webarena: A realistic web environment for building autonomous agents. arXiv preprint arXiv:2307.13854. · arXiv
  41. Zhengxia Zou, Keyan Chen, Zhenwei Shi, Yuhong Guo, and Jieping Ye. 2023. Object detection in 20 years: A survey. Proceedings of the IEEE.

Cited by

Sorted by citations

Metadata

Authors
He, H., Yao, W., Ma, K., Yu, W., Dai, Y., Zhang, H., Lan, Z., & Yu, D.
Venue
arXiv
Publisher
arXiv
Published
2024
DOI
10.48550/arxiv.2401.13919
arXiv
2401.13919
References
0
Type
Preprint
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review