CoChat
Sign Up
PreprintCited by 9Consumer Behavior in Brand Consumption and IdentificationDigital Marketing and Social MediaMarketing and Advertising Strategies

DeepShop: A Benchmark for Deep Research Shopping Agents

Open access
Lyu, Yougang, Zhang, Xiaoyu, Yan, Lingyong, de Rijke, Maarten, Ren, Zhaochun, Chen, Xiuying·2025·arXiv
DOI 10.48550/arxiv.2506.02839arXiv 2506.02839
Open this paper in CoChat — free accountOpen PDF

Abstract

Web agents for online shopping have shown great promise in automating user interactions across e-commerce platforms. Benchmarks for assessing such agents do not reflect the complexity of real-world shopping scenarios, as they often consist of overly simple queries with deterministic paths, such as "Find iPhone 15." Real shopping scenarios are inherently more layered, involving multi-dimensional product attributes, search filters, and user-specific sorting preferences. To address this gap, we introduce DeepShop, a benchmark designed to evaluate web agents in complex and realistic online shopping environments. DeepShop comprises three key components. (1) Query diversity evolution: Starting from real user queries, we generate diverse queries across five popular online shopping domains. (2) Query complexity evolution: We further evolve these queries to increase complexity, considering product attributes, search filters, and sorting preferences, and classify them into three levels: easy, medium, and hard, based on the number of evolutions. (3) Fine-grained and holistic evaluation: We propose an automated evaluation framework that assesses agent performance in terms of fine-grained aspects (product attributes, search filters, and sorting preferences) and reports the overall success rate through holistic evaluation. We conduct a systematic evaluation of retrieval-augmented generation (RAG) methods, web agents, and deep research systems. Results show that RAG struggles with complex queries due to its lack of web interaction, while other methods face significant challenges with filters and sorting preferences, leading to low overall success rates. We also perform cross-category, complexity-based evaluations and error analyses to support the advancement of deep research shopping agents.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2506.02839

Cites

1 of 1 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • The shopping agent workflow follows a common design pattern that is well-evaluated in the literature and is already productionized.

    “This shopping agent workflow follows a common design pattern that is well-evaluated in the literature [30, 47], and is already productionized [1, 21].”

    From What Is Your AI Agent Buying? Evaluation, Biases, Model Dependence, & Emerging Implications for Agentic E-Commerce (2025) · cites this paper as [30]

    The Introduction and “Web agents for task automation” section describe established web-agent approaches and prior benchmarks, supporting the point that shopping-agent workflows have been studied. But the paper says evaluations of complex, real-world shopping scenarios remain limited, so it does not establish that this workflow is already well evaluated. The production claim is attached to different citations [1, 21], not this source. (LLM, conf=0.84)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 58

  1. Abuelsaad, T., Akkil, D., Dey, P., Jagmohan, A., Vempaty, A., and Kokku, R. Agent-e: From autonomous web navigation to foundational design principles in agentic systems. arXiv preprint arXiv:2407.13032, 2024. · arXiv
  2. Cai, H., Li, Y., Wang, W., Zhu, F., Shen, X., Li, W., and Chua, T. Large language models empowered personalized web agents. CoRR, abs/2410.17236, 2024. · arXiv
  3. Chen, S., Wiseman, S., and Dhingra, B. Chatshop: Interactive information seeking with language agents. arXiv preprint arXiv:2404.09911, 2024. · arXiv
  4. Chen, Z., Choi, J. I., Fetahu, B., and Malmasi, S. Identifying high consideration e-commerce search queries. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: EMNLP 2024 -Industry Track, Miami, Florida, USA, November 12-16, 2024, pp. 563-572. Association for Computational Linguistics, 2024.
  5. Deng, X., Gu, Y., Zheng, B., Chen, S., Stevens, S., Wang, B., Sun, H., and Su, Y. Mind2web: Towards a generalist agent for the web. In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 -16, 2023, 2023.
  6. Furuta, H., Lee, K., Nachum, O., Matsuo, Y., Faust, A., Gu, S. S., and Gur, I. Multimodal web navigation with instruction-finetuned foundation models. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. Open- Review.net, 2024.
  7. Gao, S., Shi, Z., Zhu, M., Fang, B., Xin, X., Ren, P., Chen, Z., Ma, J., and Ren, Z. Confu- cius: Iterative tool learning from introspection feedback by easy-to-difficult curriculum. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 18030-18038, 2024.
  8. Gou, B., Wang, R., Zheng, B., Xie, Y., Chang, C., Shu, Y., Sun, H., and Su, Y. Navigating the digital world as humans do: Universal visual grounding for gui agents. arXiv preprint arXiv:2410.05243, 2024. · arXiv
  9. Gur, I., Furuta, H., Huang, A. V., Safdari, M., Matsuo, Y., Eck, D., and Faust, A. A real-world webagent with planning, long context understanding, and program synthesis. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024.
  10. He, H., Yao, W., Ma, K., Yu, W., Dai, Y., Zhang, H., Lan, Z., and Yu, D. Webvoyager: Building an end-to-end web agent with large multimodal models. arXiv preprint arXiv:2401.13919, 2024. · arXiv
  11. He, H., Yao, W., Ma, K., Yu, W., Zhang, H., Fang, T., Lan, Z., and Yu, D. OpenWebVoyager: Building multimodal web agents via iterative real-world exploration, feedback and optimization. arXiv preprint arXiv:2410.19609, 2024. · arXiv
  12. Hirsch, S., Guy, I., Nus, A., Dagan, A., and Kurland, O. Query reformulation in e-commerce search. In Proceedings of the 43rd International ACM SIGIR conference on research and development in Information Retrieval, SIGIR 2020, Virtual Event, China, July 25-30, 2020, pp. 1319-1328. ACM, 2020.
  13. Jang, L., Li, Y., Ding, C., Lin, J., Liang, P. P., Zhao, D., Bonatti, R., and Koishida, K. Video- WebArena: Evaluating long context multimodal agents with video understanding web tasks. arXiv preprint arXiv:2410.19100, 2024. · arXiv
  14. Kaelbling, L. P., Littman, M. L., and Cassandra, A. R. Planning and acting in partially observable stochastic domains. Artif. Intell., 101(1-2):99-134, 1998.
  15. Kanoulas, E., Eustratiadis, P., Li, Y., Lyu, Y., Pal, V., Poerwawinata, G., Qiao, J., and Wang, Z. Agent-centric information access. arXiv preprint arXiv:2502.19298, 2025. · arXiv
  16. Kim, G., Baldi, P., and McAleer, S. Language models can solve computer tasks. In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 -16, 2023, 2023.
  17. Koh, J. Y., Lo, R., Jang, L., Duvvur, V., Lim, M. C., Huang, P., Neubig, G., Zhou, S., Salakhut- dinov, R., and Fried, D. VisualWebArena: Evaluating multimodal agents on realistic visual web tasks. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2024, Bangkok, Thailand, August 11-16, 2024, pp. 881-905.
  18. Lai, H., Liu, X., Iong, I. L., Yao, S., Chen, Y., Shen, P., Yu, H., Zhang, H., Zhang, X., Dong, Y., and Tang, J. AutoWebGLM: A large language model-based web navigating agent. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, KDD 2024, Barcelona, Spain, August 25-29, 2024, pp. 5295-5306. ACM, 2024.
  19. Liu, Y., Li, P., Xie, C., Hu, X., Han, X., Zhang, S., Yang, H., and Wu, F. InfiGUI-R1: Advancing multimodal gui agents from reactive actors to deliberative reasoners. arXiv preprint arXiv:2504.14239, 2025. · arXiv
  20. Lù, X. H., Kasner, Z., and Reddy, S. Weblinx: Real-world website navigation with multi-turn dialogue. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net, 2024.
  21. Lyu, Y., Wang, Z., Ren, Z., Ren, P., Chen, Z., Liu, X., Li, Y., Li, H., and Song, H. Improving legal judgment prediction through reinforced criminal element extraction. Inf. Process. Manag., 59(1):102780, 2022.
  22. Lyu, Y., Hao, J., Wang, Z., Zhao, K., Gao, S., Ren, P., Chen, Z., Wang, F., and Ren, Z. Multi- defendant legal judgment prediction via hierarchical reasoning. In Findings of the Association for Computational Linguistics: EMNLP 2023, Singapore, December 6-10, 2023, pp. 2198-2209. Association for Computational Linguistics, 2023.
  23. Lyu, Y., Li, P., Yang, Y., de Rijke, M., Ren, P., Zhao, Y., Yin, D., and Ren, Z. Feature-level debiased natural language understanding. In Thirty-Seventh AAAI Conference on Artificial Intelligence, AAAI 2023, Thirty-Fifth Conference on Innovative Applications of Artificial Intelli- gence, IAAI 2023, Thirteenth Symposium on Educational Advances in Artificial Intelligence, EAAI 2023, Washington, DC, USA, February 7-14, 2023, pp. 13353-13361. AAAI Press, 2023.
  24. Lyu, Y., Yan, L., Wang, S., Shi, H., Yin, D., Ren, P., Chen, Z., de Rijke, M., and Ren, Z. Knowtuning: Knowledge-aware fine-tuning for large language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, EMNLP 2024, Miami, FL, USA, November 12-16, 2024, pp. 14535-14556, 2024.
  25. Lyu, Y., Yan, L., Wang, Z., Yin, D., Ren, P., de Rijke, M., and Ren, Z. MACPO: weak-to-strong alignment via multi-agent contrastive preference optimization. CoRR, abs/2410.07672, 2024. · arXiv
  26. Lyu, Y., Zhang, X., Ren, Z., and de Rijke, M. Cognitive biases in large language models for news recommendation. CoRR, abs/2410.02897, 2024. · arXiv
  27. Mialon, G., Fourrier, C., Wolf, T., LeCun, Y., and Scialom, T. GAIA: a benchmark for general AI assistants. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024.
  28. Nakano, R., Hilton, J., Balaji, S., Wu, J., Ouyang, L., Kim, C., Hesse, C., Jain, S., Kosaraju, V., Saunders, W., Jiang, X., Cobbe, K., Eloundou, T., Krueger, G., Button, K., Knight, M., Chess, B., and Schulman, J. WebGPT: Browser-assisted question-answering with human feedback. arXiv preprint arXiv:2112.09332, 2021. · arXiv
  29. Nguyen, D., Chen, J., Wang, Y., Wu, G., Park, N., Hu, Z., Lyu, H., Wu, J., Aponte, R., Xia, Y., Li, X., Shi, J., Chen, H., Lai, V. D., Xie, Z., Kim, S., Zhang, R., Yu, T., Tanjim, M. M., Ahmed, N. K., Mathur, P., Yoon, S., Yao, L., Kveton, B., Nguyen, T. H., Bui, T., Zhou, T., Rossi, R. A., and Dernoncourt, F. GUI agents: A survey. arXiv preprint arXiv:2412.13501, 2024. · arXiv
  30. Ning, L., Liang, Z., Jiang, Z., Qu, H., Ding, Y., Fan, W., Wei, X.-y., Lin, S., Liu, H., Yu, P. S., et al. A survey of webagents: Towards next-generation ai agents for web automation with large foundation models. arXiv preprint arXiv:2503.23350, 2025. · arXiv
  31. Pan, Y., Kong, D., Zhou, S., Cui, C., Leng, Y., Jiang, B., Liu, H., Shang, Y., Zhou, S., Wu, T., et al. WebCanvas: Benchmarking web agents in online environments. arXiv preprint arXiv:2406.12373, 2024. · arXiv
  32. Ren, Z., He, X., Yin, D., and de Rijke, M. Information discovery in e-commerce. Found. Trends Inf. Retr., 18(4-5):417-690, 2024.
  33. Shaw, P., Joshi, M., Cohan, J., Berant, J., Pasupat, P., Hu, H., Khandelwal, U., Lee, K., and Toutanova, K. From pixels to UI actions: Learning to follow instructions via graphical user interfaces. In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 -16, 2023, 2023.
  34. Shi, Z., Gao, S., Chen, X., Feng, Y., Yan, L., Shi, H., Yin, D., Ren, P., Verberne, S., and Ren, Z. Learning to use tools via cooperative and interactive agents. arXiv preprint arXiv:2403.03031, 2024. · arXiv
  35. Shi, Z., Sun, W., Gao, S., Ren, P., Chen, Z., and Ren, Z. Generate-then-ground in retrieval- augmented generation for multi-hop question answering. arXiv preprint arXiv:2406.14891, 2024. · arXiv
  36. Shi, Z., Gao, S., Yan, L., Feng, Y., Chen, X., Chen, Z., Yin, D., Verberne, S., and Ren, Z. Tool learning in the wild: Empowering language models as automatic tool agents. In Proceedings of the ACM on Web Conference 2025, pp. 2222-2237, 2025.
  37. Shi, Z., Yan, L., Sun, W., Feng, Y., Ren, P., Ma, X., Wang, S., Yin, D., de Rijke, M., and Ren, Z. Direct retrieval-augmented optimization: Synergizing knowledge selection and language models. arXiv preprint arXiv:2505.03075, 2025. · arXiv
  38. Shi, Z., Yan, L., Yin, D., Verberne, S., de Rijke, M., and Ren, Z. Iterative self-incentivization empowers large language models as agentic searchers. arXiv preprint arXiv:2505.20128, 2025. · arXiv
  39. Sondhi, P., Sharma, M., Kolari, P., and Zhai, C. A taxonomy of queries for e-commerce search. In The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval, SIGIR 2018, Ann Arbor, MI, USA, July 08-12, 2018, pp. 1245-1248. ACM, 2018.
  40. Song, Y., Thai, K., Pham, C. M., Chang, Y., Nadaf, M., and Iyyer, M. BEARCUBS: A benchmark for computer-using web agents. arXiv preprint arXiv:2503.07919, 2025. · arXiv
  41. Tsagkias, M., King, T. H., Kallumadi, S., Murdock, V., and de Rijke, M. Challenges and research opportunities in ecommerce search and recommendations. SIGIR Forum, 54(1):2:1-2:23, 2020.
  42. Wang, S., Liu, W., Chen, J., Gan, W., Zeng, X., Yu, S., Hao, X., Shao, K., Wang, Y., and Tang, R. GUI agents with foundation models: A comprehensive survey. CoRR, abs/2411.04890, 2024. · arXiv
  43. Wang, Z., Zhao, Z., Lyu, Y., Chen, Z., de Rijke, M., and Ren, Z. A cooperative multi- agent framework for zero-shot named entity recognition. In Proceedings of the ACM on Web Conference 2025, pp. 4183-4195, 2025.
  44. Wang, Z. Z., Gandhi, A., Neubig, G., and Fried, D. Inducing programmatic skills for agentic tasks. arXiv preprint arXiv:2504.06821, 2025. · arXiv
  45. Xu, C., Sun, Q., Zheng, K., Geng, X., Zhao, P., Feng, J., Tao, C., and Jiang, D. Wiz- ardLM: Empowering large language models to follow complex instructions. arXiv preprint arXiv:2304.12244, 2023. · arXiv
  46. Xue, T., Qi, W., Shi, T., Song, C. H., Gou, B., Song, D., Sun, H., and Su, Y. An illusion of progress? assessing the current state of web agents. arXiv preprint arXiv:2504.01382, 2025. · arXiv
  47. Yao, S., Chen, H., Yang, J., and Narasimhan, K. Webshop: Towards scalable real-world web interaction with grounded language agents. Advances in Neural Information Processing Systems, 35:20744-20757, 2022.
  48. Yoran, O., Amouyal, S. J., Malaviya, C., Bogin, B., Press, O., and Berant, J. Assistantbench: Can web agents solve realistic and time-consuming tasks? In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, EMNLP 2024, Miami, FL, USA, November 12-16, 2024, pp. 8938-8968. Association for Computational Linguistics, 2024.
  49. Zhang, C., He, S., Qian, J., Li, B., Li, L., Qin, S., Kang, Y., Ma, M., Liu, G., Lin, Q., Rajmohan, S., Zhang, D., and Zhang, Q. Large language model-brained GUI agents: A survey. arXiv preprint arXiv:2411.18279, 2024. · arXiv
  50. Zhang, H., Wang, S., Zhang, K., Tang, Z., Jiang, Y., Xiao, Y., Yan, W., and Yang, W. Towards personalized and semantic retrieval: An end-to-end solution for e-commerce search via embed- ding learning. In Proceedings of the 43rd International ACM SIGIR conference on research and development in Information Retrieval, SIGIR 2020, Virtual Event, China, July 25-30, 2020, pp. 2407-2416. ACM, 2020.
  51. Zhang, X., Xin, X., Li, D., Liu, W., Ren, P., Chen, Z., Ma, J., and Ren, Z. Variational reasoning over incomplete knowledge graphs for conversational recommendation. In Proceedings of the Sixteenth ACM International Conference on Web Search and Data Mining, WSDM 2023, Singapore, 27 February 2023 -3 March 2023, pp. 231-239. ACM, 2023.
  52. Zhang, X., Xie, R., Lyu, Y., Xin, X., Ren, P., Liang, M., Zhang, B., Kang, Z., de Rijke, M., and Ren, Z. Towards empathetic conversational recommender systems. In Proceedings of the 18th ACM Conference on Recommender Systems, RecSys 2024, Bari, Italy, October 14-18, 2024, pp. 84-93. ACM, 2024.
  53. Zhang, X., Xie, R., Lyu, Y., Xin, X., Ren, P., Liang, M., Zhang, B., Kang, Z., de Rijke, M., and Ren, Z. Towards empathetic conversational recommender systems. arXiv preprint arXiv:2409.10527, 2024. · arXiv
  54. Zhang, Z., Tian, S., Chen, L., and Liu, Z. MMInA: Benchmarking multihop multimodal internet agents. arXiv preprint arXiv:2404.09992, 2024. · arXiv
  55. Zhao, Z., Ren, Z., Yang, J., Yan, Z., Wang, Z., Yang, L., Ren, P., Chen, Z., de Rijke, M., and Xin, X. Improving sequential recommenders through counterfactual augmentation of system exposure. arXiv preprint arXiv:2504.13482, 2025. · arXiv
  56. Zheng, B., Gou, B., Kil, J., Sun, H., and Su, Y. Gpt-4v(ision) is a generalist web agent, if grounded. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net, 2024.
  57. Zheng, B., Fatemi, M. Y., Jin, X., Wang, Z. Z., Gandhi, A., Song, Y., Gu, Y., Srinivasa, J., Liu, G., Neubig, G., et al. SkillWeaver: Web agents can self-improve by discovering and honing skills. arXiv preprint arXiv:2504.07079, 2025. · arXiv
  58. Zhou, S., Xu, F. F., Zhu, H., Zhou, X., Lo, R., Sridhar, A., Cheng, X., Ou, T., Bisk, Y., Fried, D., Alon, U., and Neubig, G. WebArena: A realistic web environment for building autonomous agents. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024.

Cited by

Sorted by citations

Authors
Lyu, Y., Zhang, X., Yan, L., de Rijke, M., Ren, Z., & Chen, X.
Affiliations
  • University of Amsterdam (NL)
  • Shandong University (CN)
  • Baidu (China) (CN)
  • Leiden University (NL)
  • Mohamed bin Zayed University of Artificial Intelligence (AE)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2025
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2506.02839
arXiv
2506.02839
OpenAlex
W4414899318
Cited by
9
References
1
Citations by year
2026: 8 · 2025: 1
Type
posted content
Language
en
Open access
Licence
Locations
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University)
  • Zenodo (CERN European Organization for Nuclear Research) (cc-by)
Funders
  • European Commission (101070212)
  • National Natural Science Foundation of China (62102234, 62272274, 62202271, 62072279)
  • Nederlandse Organisatie voor Wetenschappelijk Onderzoek (024.004.022, NWA.1389.20.183, KICH3.LTP.20.006)
  • Natural Science Foundation of Shandong Province (ZR2021QF129)
  • HORIZON EUROPE Framework Programme
Topics
  • Social Sciences › Business, Management and Accounting › Marketing › Consumer Behavior in Brand Consumption and Identification
  • Social Sciences › Sociology and Political Science › Digital Marketing and Social Media
  • Social Sciences › Business, Management and Accounting › Marketing › Marketing and Advertising Strategies
Keywords
brand identity, AI agents, brand functions, cryptographic verification, brand equity, brand coherence, computational trust, behavioral contracts, adverse selection, moral hazard

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review