CoChat
Sign Up
PreprintCited by 22Multimodal Machine Learning ApplicationsWeb Data Mining and AnalysisInformation Retrieval and Search Behavior

GPT-4V(ision) is a Generalist Web Agent, if Grounded

Open access
Zheng, Boyuan, Gou, Boyu, Kil, Jihyung, Sun, Huan, Su, Yu·2024·arXiv
DOI 10.48550/arxiv.2401.01614arXiv 2401.01614
Open this paper in CoChat — free accountOpen PDF

Abstract

The recent development on large multimodal models (LMMs), especially GPT-4V(ision) and Gemini, has been quickly expanding the capability boundaries of multimodal models beyond traditional tasks like image captioning and visual question answering. In this work, we explore the potential of LMMs like GPT-4V as a generalist web agent that can follow natural language instructions to complete tasks on any given website. We propose SEEACT, a generalist web agent that harnesses the power of LMMs for integrated visual understanding and acting on the web. We evaluate on the recent MIND2WEB benchmark. In addition to standard offline evaluation on cached websites, we enable a new online evaluation setting by developing a tool that allows running web agents on live websites. We show that GPT-4V presents a great potential for web agents -- it can successfully complete 51.1 of the tasks on live websites if we manually ground its textual plans into actions on the websites. This substantially outperforms text-only LLMs like GPT-4 or smaller models (FLAN-T5 and BLIP-2) specifically fine-tuned for web agents. However, grounding still remains a major challenge. Existing LMM grounding strategies like set-of-mark prompting turns out to be not effective for web agents, and the best grounding strategy we develop in this paper leverages both the HTML structure and visuals. Yet, there is still a substantial gap with oracle grounding, leaving ample room for further improvement. All code, data, and evaluation tools are available at https://github.com/OSU-NLP-Group/SeeAct.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2401.01614

Cites

1 of 1 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Code-powered agents and web agents are popular agent types, and both healthcare and online shopping are typical domains that involve highly sensitive user private information.

    “Code-powered agents and web agents are popular agent types (Wang et al., 2024b; Trivedi et al., 2024; Zheng et al., 2024; Deng et al., 2023), and both healthcare and online shopping are typical domains that involve highly sensitive user private information.”

    From Unveiling Privacy Risks in LLM Agent Memory (2025) · cites this paper as Zheng et al., 2024

    The Related Work section describes many studies of web agents, supporting that part of the cited clause. The passages do not establish that code-powered agents are a popular agent type. The healthcare and online-shopping statement follows the citation and is not attributed to this source by its placement. (LLM, conf=0.85)

    Experimental — accuracy not yet measured

  • SeeAct is a state-of-the-art web agent framework.

    “SeeAct (Zheng et al., 2024) proposes a generalist web agent framework featuring a two-stage pipeline that incorporates rendered webpage screenshots as input, significantly improving reasoning and achieving state-of-the-art task completion performance.”

    From AdvAgent: Controllable Blackbox Red-teaming on Web Agents (2024) · cites this paper as Zheng et al., 2024

  • SeeAct is a state-of-the-art web agent powered by different proprietary VLMs.

    “To demonstrate the effectiveness of AdvAgent, we employ SeeAct (Zheng et al., 2024), a state-of-the-art web agent powered by different proprietary VLMs (Achiam et al., 2023;Team et al., 2023).”

    From AdvAgent: Controllable Blackbox Red-teaming on Web Agents (2024) · cites this paper as Zheng et al., 2024

  • SeeAct operates by first generating an action description based on the current task and the webpage screenshot, then mapping this description to the corresponding HTML contents to interact with the web environment.

    “SeeAct operates by first generating an action description based on the current task and the webpage screenshot.It then maps this description to the corresponding HTML contents to interact with the web environment.”

    From AdvAgent: Controllable Blackbox Red-teaming on Web Agents (2024) · cites this paper as Zheng et al., 2024

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 47

  1. Akter, S. N., Yu, Z., Muhamed, A., Ou, T., Bauerle, A., Cabrera, Á. A., Dholakia, K., Xiong, C., and Neubig, G. An in-depth look at gemini's language abilities. 2023. URL https://api.semanticscholar. org/CorpusID:266359502.
  2. Alayrac, J.-B., Donahue, J., Luc, P., Miech, A., Barr, I., Hasson, Y., Lenc, K., Mensch, A., Millican, K., Reynolds, M., Ring, R., Rutherford, E., Cabi, S., Han, T., Gong, Z., Samangooei, S., Monteiro, M., Menick, J., Borgeaud, S., Brock, A., Nematzadeh, A., Sharifzadeh, S., Binkowski, M., Barreira, R., Vinyals, O., Zisserman, A., and Simonyan, K. Flamingo: a visual language model for few-shot learning. ArXiv, abs/2204.14198, 2022. URL https://api.semanticscholar. org/CorpusID:248476411. · arXiv
  3. Anil, G. T. G. R., Borgeaud, S., and et al., Y. W. Gem- ini: A family of highly capable multimodal models. 2023. URL https://api.semanticscholar. org/CorpusID:266361876.
  4. Chandu, K. R., Bisk, Y., and Black, A. W. Grounding 'grounding'in nlp. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pp. 4283- 4305, 2021.
  5. Chen, K., Zhang, Z., Zeng, W., Zhang, R., Zhu, F., and Zhao, R. Shikra: Unleashing multimodal llm's referential dialogue magic. ArXiv, abs/2306.15195, 2023. URL https://api.semanticscholar. org/CorpusID:259262082. · arXiv
  6. Cheng, K., Sun, Q., Chu, Y., Xu, F., Li, Y., Zhang, J., and Wu, Z. Seeclick: Harness- ing gui grounding for advanced visual gui agents. 2024. URL https://api.semanticscholar. org/CorpusID:267069082.
  7. Chung, H. W., Hou, L., Longpre, S., Zoph, B., Tay, Y., Fedus, W., Li, E., Wang, X., Dehghani, M., Brahma, S., Webson, A., Gu, S. S., Dai, Z., Suzgun, M., Chen, X., Chowdhery, A., Valter, D., Narang, S., Mishra, G., Yu, A. W., Zhao, V., Huang, Y., Dai, A. M., Yu, H., Petrov, S., hsin Chi, E. H., Dean, J., Devlin, J., Roberts, A., Zhou, D., Le, Q. V., and Wei, J. Scaling instruction- finetuned language models. ArXiv, abs/2210.11416, 2022. URL https://api.semanticscholar. org/CorpusID:253018554. · arXiv
  8. Deng, X., Gu, Y., Zheng, B., Chen, S., Stevens, S., Wang, B., Sun, H., and Su, Y. Mind2web: Towards a generalist agent for the web. arXiv preprint arXiv:2306.06070, 2023. · arXiv
  9. Furuta, H., Nachum, O., Lee, K.-H., Matsuo, Y., Gu, S. S., and Gur, I. Multimodal web navigation with instruction- finetuned foundation models. ArXiv, abs/2305.11854, 2023. URL https://api.semanticscholar. org/CorpusID:258823350. · arXiv
  10. Goyal, Y., Khot, T., Summers-Stay, D., Batra, D., and Parikh, D. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. International Journal of Computer Vision, 127:398 -414, 2016. URL https://api.semanticscholar. org/CorpusID:8081284.
  11. Gu, Y., Deng, X., and Su, Y. Don't generate, discrimi- nate: A proposal for grounding language models to real- world environments. In Rogers, A., Boyd-Graber, J., and Okazaki, N. (eds.), Proceedings of the 61st Annual Meet- ing of the Association for Computational Linguistics (Vol- ume 1: Long Papers), pp. 4928-4949, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023.acl-long.270. URL https: //aclanthology.org/2023.acl-long.270. · DOI
  12. Guan, T., Liu, F., Wu, X., Xian, R., Li, Z., Liu, X., Wang, X., Chen, L., Huang, F., Yacoob, Y., Manocha, D., and Zhou, T. Hallusionbench: An ad- vanced diagnostic suite for entangled language halluci- nation&visual illusion in large vision-language models. 2023. URL https://api.semanticscholar. org/CorpusID:265499116.
  13. Gur, I., Nachum, O., Miao, Y., Safdari, M., Huang, A., Chowdhery, A., Narang, S., Fiedel, N., and Faust, A. Understanding html with large language models. In Conference on Empirical Methods in Natural Language Processing, 2022. URL https: //api.semanticscholar.org/CorpusID: 252780086.
  14. Gur, I., Furuta, H., Huang, A., Safdari, M., Mat- suo, Y., Eck, D., and Faust, A. A real-world webagent with planning, long context understand- ing, and program synthesis. ArXiv, abs/2307.12856, 2023. URL https://api.semanticscholar. org/CorpusID:260126067. · arXiv
  15. Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D. X., and Steinhardt, J. Measuring massive multi- task language understanding. ArXiv, abs/2009.03300, 2020. URL https://api.semanticscholar. org/CorpusID:221516475. · arXiv
  16. Hong, W., Wang, W., Lv, Q., Xu, J., Yu, W., Ji, J., Wang, Y., Wang, Z., Dong, Y., Ding, M., and Tang, J. Cogagent: A visual language model for gui agents. 2023. URL https://api.semanticscholar. org/CorpusID:266210390.
  17. Kazemzadeh, S., Ordonez, V., andre Matten, M., and Berg, T. L. Referitgame: Referring to objects in photographs of natural scenes. In Conference on Empirical Methods in Natural Language Processing, 2014. URL https://api.semanticscholar. org/CorpusID:6308361.
  18. Kim, G., Baldi, P., and McAleer, S. M. Language mod- els can solve computer tasks. ArXiv, abs/2303.17491, 2023. URL https://api.semanticscholar. org/CorpusID:257834038. · arXiv
  19. Koh, J. Y., Lo, R., Jang, L., Duvvur, V., Lim, M. C., Huang, P.-Y., Neubig, G., Zhou, S., Salakhutdi- nov, R., and Fried, D. Visualwebarena: Evaluat- ing multimodal agents on realistic visual web tasks. 2024. URL https://api.semanticscholar. org/CorpusID:267199749.
  20. Lee, K., Joshi, M., Turc, I., Hu, H., Liu, F., Eisen- schlos, J. M., Khandelwal, U., Shaw, P., Chang, M.-W., and Toutanova, K. Pix2struct: Screenshot parsing as pretraining for visual language understand- ing. ArXiv, abs/2210.03347, 2022. URL https: //api.semanticscholar.org/CorpusID: 252762394. · arXiv
  21. Li, J., Li, D., Savarese, S., and Hoi, S. C. H. Blip- 2: Bootstrapping language-image pre-training with frozen image encoders and large language mod- els. ArXiv, abs/2301.12597, 2023. URL https: //api.semanticscholar.org/CorpusID: 256390509. · arXiv
  22. Lin, T.-Y., Maire, M., Belongie, S. J., Hays, J., Perona, P., Ramanan, D., Dollár, P., and Zitnick, C. L. Microsoft coco: Common objects in context. In European Confer- ence on Computer Vision, 2014. URL https://api. semanticscholar.org/CorpusID:14113767.
  23. Liu, E. Z., Guu, K., Pasupat, P., Shi, T., and Liang, P. Rein- forcement learning on web interfaces using workflow- guided exploration. In International Conference on Learning Representations (ICLR), 2018. URL https: //arxiv.org/abs/1802.08802.
  24. Liu, H., Li, C., Li, Y., and Lee, Y. J. Improved baselines with visual instruction tuning, 2023a. Liu, H., Li, C., Wu, Q., and Lee, Y. J. Vi- sual instruction tuning. ArXiv, abs/2304.08485, 2023b. URL https://api.semanticscholar. org/CorpusID:258179774. · arXiv
  25. Liu, H., Li, C., Wu, Q., and Lee, Y. J. Visual instruction tuning, 2023c. Liu, X., Yu, H., Zhang, H., Xu, Y., Lei, X., Lai, H., Gu, Y., Gu, Y., Ding, H., Men, K., Yang, K., Zhang, S., Deng, X., Zeng, A., Du, Z., Zhang, C., Shen, S., Zhang, T., Su, Y., Sun, H., Huang, M., Dong, Y., and Tang, J. Agent- bench: Evaluating llms as agents. ArXiv, abs/2308.03688, 2023d. URL https://api.semanticscholar. org/CorpusID:260682249. · arXiv
  26. Lu, P., Mishra, S., Xia, T., Qiu, L., Chang, K.-W., Zhu, S.-C., Tafjord, O., Clark, P., and Kalyan, A. Learn to explain: Multimodal reasoning via thought chains for science question answering. ArXiv, abs/2209.09513, 2022. URL https://api.semanticscholar. org/CorpusID:252383606. · arXiv
  27. Mazumder, S. and Riva, O. Flin: A flexible natural language interface for web navigation. ArXiv, abs/2010.12844, 2020. URL https://api.semanticscholar. org/CorpusID:225067907. · arXiv
  28. OpenAI. Gpt-4 technical report. ArXiv, abs/2303.08774, 2023. URL https://api.semanticscholar. org/CorpusID:257532815. · arXiv
  29. Peng, Z., Wang, W., Dong, L., Hao, Y., Huang, S., Ma, S., and Wei, F. Kosmos-2: Grounding multimodal large language models to the world. ArXiv, abs/2306.14824, 2023. URL https://api.semanticscholar. org/CorpusID:259262263. · arXiv
  30. Plummer, B. A., Wang, L., Cervantes, C. M., Caicedo, J. C., Hockenmaier, J., and Lazebnik, S. Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models. International Journal of Com- puter Vision, 123:74 -93, 2015. URL https://api. semanticscholar.org/CorpusID:6941275.
  31. Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748-8763. PMLR, 2021.
  32. Saikh, T., Ghosal, T., Mittal, A., Ekbal, A., and Bhattacharyya, P. Scienceqa: a novel resource for question answering on scholarly articles. Interna- tional Journal on Digital Libraries, 23:289 -301, 2022. URL https://api.semanticscholar. org/CorpusID:250729995.
  33. Shaw, P., Joshi, M., Cohan, J., Berant, J., Pasupat, P., Hu, H., Khandelwal, U., Lee, K., and Toutanova, K. From pixels to ui actions: Learning to follow instructions via graphical user interfaces. ArXiv, abs/2306.00245, 2023. URL https://api.semanticscholar. org/CorpusID:258999511. · arXiv
  34. Shen, S., Li, L. H., Tan, H., Bansal, M., Rohrbach, A., Chang, K.-W., Yao, Z., and Keutzer, K. How much can clip benefit vision-and-language tasks? arXiv preprint arXiv:2107.06383, 2021. · arXiv
  35. Shi, T., Karpathy, A., Fan, L. J., Hernández, J. Z., and Liang, P. World of bits: An open-domain platform for web-based agents. In International Conference on Machine Learning, 2017. URL https://api. semanticscholar.org/CorpusID:34953552.
  36. Shtedritski, A., Rupprecht, C., and Vedaldi, A. What does clip know about a red circle? visual prompt engineering for vlms. ArXiv, abs/2304.06712, 2023. URL https://api.semanticscholar. org/CorpusID:258108138. · arXiv
  37. Sridhar, A., Lo, R., Xu, F. F., Zhu, H., and Zhou, S. Hierarchical prompting assists large language model on web navigation. ArXiv, abs/2305.14257, 2023. URL https://api.semanticscholar. org/CorpusID:258841249. · arXiv
  38. Yan, A., Yang, Z., Zhu, W., Lin, K., Li, L., Wang, J., Yang, J., Zhong, Y., McAuley, J., Gao, J., Liu, Z., and Wang, L. Gpt-4v in wonderland: Large multi- modal models for zero-shot smartphone gui navigation. 2023. URL https://api.semanticscholar. org/CorpusID:265149992.
  39. Yang, J., Zhang, H., Li, F., Zou, X., yue Li, C., and Gao, J. Set-of-mark prompting unleashes extraordi- nary visual grounding in gpt-4v. ArXiv, abs/2310.11441, 2023a. URL https://api.semanticscholar. org/CorpusID:264172201. · arXiv
  40. Yang, L., Wang, Y., Li, X., Wang, X., and Yang, J. Fine-grained visual prompting. ArXiv, abs/2306.04356, 2023b. URL https://api.semanticscholar. org/CorpusID:259096008. · arXiv
  41. Yang, Z., Li, L., Lin, K., Wang, J., Lin, C.-C., Liu, Z., and Wang, L. The dawn of lmms: Preliminary ex- plorations with gpt-4v(ision). ArXiv, abs/2309.17421, 2023c. URL https://api.semanticscholar. org/CorpusID:263310951. · arXiv
  42. Yao, S., Chen, H., Yang, J., and Narasimhan, K. Web- shop: Towards scalable real-world web interaction with grounded language agents. ArXiv, abs/2207.01206, 2022. URL https://api.semanticscholar. org/CorpusID:250264533. · arXiv
  43. You, H., Zhang, H., Gan, Z., Du, X., Zhang, B., Wang, Z., Cao, L., Chang, S.-F., and Yang, Y. Ferret: Refer and ground anything any- where at any granularity. ArXiv, abs/2310.07704, 2023. URL https://api.semanticscholar. org/CorpusID:263834718. · arXiv
  44. Yue, X., Ni, Y., Zhang, K., Zheng, T., Liu, R., Zhang, G., Stevens, S., Jiang, D., Ren, W., Sun, Y., Wei, C., Yu, B., Yuan, R., Sun, R., Yin, M., Zheng, B., Yang, Z., Liu, Y., Huang, W., Sun, H., Su, Y., and Chen, W. Mmmu: A mas- sive multi-discipline multimodal understanding and rea- soning benchmark for expert agi. ArXiv, abs/2311.16502, 2023. URL https://api.semanticscholar. org/CorpusID:265466525. · arXiv
  45. Zhang, X., Lu, Y., Wang, W., Yan, A., Yan, J., Qin, L., Wang, H., Yan, X., Wang, W. Y., and Pet- zold, L. R. Gpt-4v(ision) as a generalist evaluator for vision-language tasks. ArXiv, abs/2311.01361, 2023. URL https://api.semanticscholar. org/CorpusID:264935635. · arXiv
  46. Zhao, Y., Lin, Z., Zhou, D., Huang, Z., Feng, J., and Kang, B. Bubogpt: Enabling visual ground- ing in multi-modal llms. ArXiv, abs/2307.08581, 2023. URL https://api.semanticscholar. org/CorpusID:259937702. · arXiv
  47. Zhou, B., Zhao, H., Puig, X., Fidler, S., Barriuso, A., and Torralba, A. Scene parsing through ade20k dataset. 2017 IEEE Conference on Computer Vi- sion and Pattern Recognition (CVPR), pp. 5122-5130, 2017. URL https://api.semanticscholar. org/CorpusID:5636055.

Cited by

Sorted by citations

Authors
Zheng, B., Gou, B., Kil, J., Sun, H., & Su, Y.
ORCID
Venue
arXiv
Publisher
arXiv
Published
2024
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2401.01614
arXiv
2401.01614
OpenAlex
W4390601403
Cited by
10
References
1
Citations by year
2026: 2 · 2025: 3 · 2024: 5
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Funders
Cisco Systems
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
  • Physical Sciences › Computer Science › Information Systems › Web Data Mining and Analysis
  • Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
Keywords
GPT-4V, multimodal large language models, web agents, grounded language models, HTML structure, visual grounding, natural language instructions

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review