CoChat
Sign Up
Cited by 8Information Retrieval and Search BehaviorNatural Language Processing TechniquesTopic Modeling

Dataset of Natural Language Queries for E-Commerce

Open access
Papenmeier, Andrea, Kern, Dagmar, Hienert, Daniel, Sliwa, Alfred, Aker, Ahmet, Fuhr, Norbert·2021·Proceedings of the 2021 Conference on Human Information Interaction and Retrieval
DOI 10.1145/3406522.3446043
Open this paper in CoChat — free accountOpen PDF

Abstract

Shopping online is more and more frequent in our everyday life. For e-commerce search systems, understanding natural language coming through voice assistants, chatbots or from conversational search is an essential ability to understand what the user really wants. However, evaluation datasets with natural and detailed information needs of product-seekers which could be used for research do not exist. Due to privacy issues and competitive consequences, only few datasets with real user search queries from logs are openly available. In this paper, we present a dataset of 3,540 natural language queries in two domains that describe what users want when searching for a laptop or a jacket of their choice. The dataset contains annotations of vague terms and key facts of 1,754 laptop queries. This dataset opens up a range of research opportunities in the fields of natural language processing and (interactive) information retrieval for product search.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1145/3406522.3446043

Cites

1 of 1 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • WANDS and ESCI average ∼3-3.5 words in a single sentence and Papenmeier et al. (2021)'s queries average ∼33 words, while E-GEO queries average ∼59 words.

    “Quantitatively, WANDS (Chen et al., 2022) and ESCI (Reddy et al., 2022) average ∼ 3-3.5 words in a single sentence and Papenmeier et al. (2021)'s queries average ∼ 33 words , while E-GEO queries average ∼ 59 words (Table 2, which also lists representative example queries).”

    From E-GEO: A Testbed for Generative Engine Optimization in E-Commerce (2025) · cites this paper as Papenmeier et al., 2021

    Table 2 (Dataset Statistics; page not available) reports a mean of 35 words per query for the laptop corpus, close to but not the cited ∼33. Our copy does not give a 33-word mean for Papenmeier et al.’s queries as a whole. The WANDS, ESCI, and E-GEO figures are not attributed to this source. (LLM, conf=0.84)

    Experimental — accuracy not yet measured

  • Existing e-commerce datasets target retrieval relevance and use short, keyword-style queries, which do not resemble realistic interactions with shopping agents.

    “Existing e-commerce datasets target retrieval relevance and use short, keyword-style queries (Papenmeier et al., 2021; Chen et al., 2022; Reddy et al., 2022), which do not resemble realistic interactions with shopping agents.”

    From E-GEO: A Testbed for Generative Engine Optimization in E-Commerce (2025) · cites this paper as Papenmeier et al., 2021

    Section 2.2 says publicly available query datasets contain keyword queries rather than users’ natural information needs, and the Introduction explains why keyword searches may not reflect how users would express those needs to a conversational system. But the paper describes most existing e-commerce datasets as focusing on catalogues, taxonomies, reviews, and recommendations; it does not establish that they generally target retrieval relevance or that their queries are short. Page numbers are unavailable in our copy. (LLM, conf=0.85)

    Experimental — accuracy not yet measured

  • The dataset provides both breadth (a large number of queries, in three languages) and depth (approximately 20 results per query), unlike other existing large document retrieval collections which tend to provide either breadth or depth but not both.

    “Second, it provides both breadth (a large number of queries, in three languages) and depth ( ≈ 20 results per query), unlike other existing large document retrieval collections which tend to provide either breadth or depth but not both (e.g. MSMarco [2], TRECDL [6], NLQEC [13]).”

    From Shopping Queries Dataset: A Large-Scale ESCI Benchmark for Improving Product Search (2022) · cites this paper as [13]

    Split into 2 atoms — 1 confirmed, 1 partially supported, 0 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.

    Experimental — accuracy not yet measured

  • Aspect-value extraction identifies portions of a string of text (values) for more narrowly defined aspects like brand or color.

    “Aspect-value extraction identifies portions of a string of text (values) for more narrowly defined aspects like brand or color (Joshi et al. 2015;Papenmeier et al. 2021).”

    From QueryNER: Segmentation of E-commerce Queries (2024) · cites this paper as Papenmeier et al. 2021

    Section 3.2 (page unavailable) describes labeling words or phrases in product queries as key facts, and Section 4.2 discusses identifying their attribute domains. This supports extracting text spans associated with product attributes, but the paper does not define aspect-value extraction or give brand and color as examples of extracted aspect-value pairs. (LLM, conf=0.82)

    Experimental — accuracy not yet measured

  • Chunking is often framed as a sequence labeling task, and while there has been sequence labeling work in e-commerce, it has largely focused on aspect-value extraction.

    “Chunking is often framed as a sequence labeling task, and while there has been sequence labeling work in e-commerce, it has largely focused on aspect-value extraction.”

    From QueryNER: Segmentation of E-commerce Queries (2024) · cites this paper as Papenmeier et al. 2021

    Section 3.2 (page unavailable) describes an e-commerce query annotation task conducted as sequence labelling, supporting the existence of such work. Neither that section nor §4.2 says that e-commerce sequence-labelling work has largely focused on aspect-value extraction; this paper labels key facts and vague words instead. (LLM, conf=0.86)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 30

  1. Mohammad Aliannejadi, Hamed Zamani, Fabio Crestani, and W. Bruce Croft. 2019. Asking Clarifying Questions in Open-Domain Information-Seeking Conversations. In Proceedings of the 42nd International ACM SIGIR Confer- ence on Research and Development in Information Retrieval (Paris, France) (SI- GIR'19). Association for Computing Machinery, New York, NY, USA, 475-484. https://doi.org/10.1145/3331184.3331265 · DOI
  2. Peter Bailey, Alistair Moffat, Falk Scholer, and Paul Thomas. 2015. User Vari- ability and IR System Evaluation. In Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval (Santi- ago, Chile) (SIGIR '15). Association for Computing Machinery, New York, NY, USA, 625-634. https://doi.org/10.1145/2766462.2767728 · DOI
  3. Toine Bogers, Maria Gäde, Marijn Koolen, Vivien Petras, and Mette Skov. 2018. "What was this Movie About this Chick?". In Transforming Digital Worlds, Gob- inda Chowdhury, Julie McLeod, Val Gillet, and Peter Willett (Eds.). Springer In- ternational Publishing, Cham, 323-334.
  4. Andrei Broder. 2002. A Taxonomy of Web Search. SIGIR Forum 36, 2 (Sept. 2002), 3-10. https://doi.org/10.1145/792550.792552 · DOI · Paper page
  5. Layla El Asri, Hannes Schulz, Shikhar Kr Sarma, Jeremie Zumer, Justin Harris, Emery Fine, Rahul Mehrotra, and Kaheer Suleman. 2017. Frames: a corpus for adding memory to goal-oriented dialogue systems. In Proceedings of the 18th Annual SIGdial Meeting on Discourse and Dialogue. 207-219.
  6. George W. Furnas, Thomas K. Landauer, Louis M. Gomez, and Susan T. Dumais. 1987. The vocabulary problem in human-system communication. Commun. ACM 30, 11 (1987), 964-971.
  7. Jiafeng Guo, Gu Xu, Xueqi Cheng, and Hang Li. 2009. Named Entity Recog- nition in Query. In Proceedings of the 32nd International ACM SIGIR Conference on Research and Development in Information Retrieval (Boston, MA, USA) (SI- GIR '09). Association for Computing Machinery, New York, NY, USA, 267-274. https://doi.org/10.1145/1571941.1571989 · DOI
  8. Yvonne Kammerer and Maja Bohnacker. 2012. Children's Web Search with Google: The Effectiveness of Natural Language Queries. In Proceedings of the 11th International Conference on Interaction Design and Children (Bremen, Ger- many) (IDC '12). Association for Computing Machinery, New York, NY, USA, 184-187. https://doi.org/10.1145/2307096.2307121 · DOI
  9. Makoto P. Kato, Takehiro Yamamoto, Hiroaki Ohshima, and Katsumi Tanaka. 2014. Cognitive Search Intents Hidden behind Queries: A User Study on Query Formulations. In Proceedings of the 23rd International Conference on World Wide Web (Seoul, Korea) (WWW '14 Companion). Association for Computing Machin- ery, New York, NY, USA, 313-314. https://doi.org/10.1145/2567948.2577279 · DOI
  10. Marijn Koolen, Toine Bogers, Maria Gäde, Mark Hall, Iris Hendrickx, Hugo Huurdeman, Jaap Kamps, Mette Skov, Suzan Verberne, and David Walsh. 2016. Overview of the CLEF 2016 social book search lab. In International conference of the cross-language evaluation forum for European languages. Springer, 351-370.
  11. Felipe Moraes, Jie Yang, Rongting Zhang, and Vanessa Murdock. 2020. The Role of Attributes in Product Quality Comparisons. In Proceedings of the 2020 Con- ference on Human Information Interaction and Retrieval (Vancouver BC, Canada) (CHIIR '20). Association for Computing Machinery, New York, NY, USA, 253-262. https://doi.org/10.1145/3343413.3377956 · DOI
  12. Andrea Papenmeier, Alfred Sliwa, Dagmar Kern, Daniel Hienert, Ahmet Aker, and Norbert Fuhr. 2020. 'A Modern Up-To-Date Laptop'-Vagueness in Natural Language Queries for Product Search. In Proceedings of the 2020 ACM Designing Interactive Systems Conference. 2077-2089.
  13. Gustavo Penha, Alexandru Balan, and Claudia Hauff. 2019. Introducing MAN- tIS: a novel multi-domain information seeking dialogues dataset. arXiv preprint arXiv:1912.04639 (2019). · arXiv
  14. Chen Qu, Liu Yang, W. Bruce Croft, Johanne R. Trippas, Yongfeng Zhang, and Minghui Qiu. 2018. Analyzing and Characterizing User Intent in Information- Seeking Conversations (SIGIR '18). Association for Computing Machinery, New York, NY, USA, 989-992. https://doi.org/10.1145/3209978.3210124 · DOI
  15. Filip Radlinski, Krisztian Balog, Bill Byrne, and Karthik Krishnamoorthi. 2019. Coached Conversational Preference Elicitation: A Case Study in Understanding Movie Preferences. In Proceedings of the Annual SIGdial Meeting on Discourse and Dialogue.
  16. Eldar Sadikov, Jayant Madhavan, Lu Wang, and Alon Halevy. 2010. Clus- tering Query Refinements by User Intent. In Proceedings of the 19th Interna- tional Conference on World Wide Web (Raleigh, North Carolina, USA) (WWW '10). Association for Computing Machinery, New York, NY, USA, 841-850. https://doi.org/10.1145/1772690.1772776 · DOI
  17. Yelong Shen, Jun Yan, Shuicheng Yan, Lei Ji, Ning Liu, and Zheng Chen. 2011. Sparse Hidden-Dynamics Conditional Random Fields for User Intent Under- standing. In Proceedings of the 20th International Conference on World Wide Web (Hyderabad, India) (WWW '11). Association for Computing Machinery, New York, NY, USA, 7-16. https://doi.org/10.1145/1963405.1963411 · DOI
  18. Michael Skinner and Surya Kallumadi. 2019. E-commerce Query Classification Using Product Taxonomy Mapping: A Transfer Learning Approach.. In eCOM@ SIGIR.
  19. Parikshit Sondhi, Mohit Sharma, Pranam Kolari, and ChengXiang Zhai. 2018. A Taxonomy of Queries for E-Commerce Search. In The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval (Ann Arbor, MI, USA) (SIGIR '18). Association for Computing Machinery, New York, NY, USA, 1245-1248. https://doi.org/10.1145/3209978.3210152 · DOI
  20. Daria Sorokina and Erick Cantu-Paz. 2016. Amazon search: The joy of rank- ing products. In Proceedings of the 39th International ACM SIGIR conference on Research and Development in Information Retrieval. 459-460.
  21. Ning Su, Jiyin He, Yiqun Liu, Min Zhang, and Shaoping Ma. 2018. User Intent, Behaviour, and Perceived Satisfaction in Product Search. In Proceedings of the Eleventh ACM International Conference on Web Search and Data Mining (Marina Del Rey, CA, USA) (WSDM '18). Association for Computing Machinery, New York, NY, USA, 547-555. https://doi.org/10.1145/3159652.3159714 · DOI
  22. Paul Thomas, Daniel McDuff, Mary Czerwinski, and Nick Craswell. 2017. MISC: A data set of information-seeking conversations. In SIGIR 1st International Work- shop on Conversational Approaches to Information Retrieval (CAIR'17), Vol. 5.
  23. Johanne R. Trippas, Damiano Spina, Lawrence Cavedon, and Mark Sanderson. 2017. How Do People Interact in Conversational Speech-Only Search Tasks: A Preliminary Analysis. In Proceedings of the 2017 Conference on Conference Hu- man Information Interaction and Retrieval (Oslo, Norway) (CHIIR '17). ACM, New York, NY, USA, 325-328. https://doi.org/10.1145/3020165.3022144 · DOI
  24. Andrew Trotman, Jon Degenhardt, and Surya Kallumadi. 2017. The Archi- tecture of eBay Search. In Proceedings of the SIGIR 2017 Workshop On eCom- merce co-located with the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval, eCOM@SIGIR 2017, Tokyo, Japan, Au- gust 11, 2017 (CEUR Workshop Proceedings, Vol. 2311), Jon Degenhardt, Surya Kallumadi, Maarten de Rijke, Luo Si, Andrew Trotman, and Yinghui Xu (Eds.). CEUR-WS.org. http://ceur-ws.org/Vol-2311/paper_14.pdf
  25. Manos Tsagkias, Tracy Holloway King, Surya Kallumadi, Vanessa Murdock, and Maarten de Rijke. 2020. Challenges and Research Opportunities in eCommerce Search and Recommendations. In SIGIR Forum, Vol. 54.
  26. Ellen M. Voorhees and Donna K. Harman. 2005. TREC: Experiment and Evalu- ation in Information Retrieval (Digital Libraries and Electronic Publishing). The MIT Press.
  27. Chao Wang and Rongkai Zhao. 2019. Multi-Candidate Ranking Algorithm Based Spell Correction.. In Proceedings of the SIGIR 2019 Workshop On eCommerce co- located with the 42st International ACM SIGIR Conference on Research and Devel- opment in Information Retrieval (SIGIR 2019) Paris, France, July 25, 2019. (CEUR Workshop Proceedings), Jon Degenhardt, Surya Kallumadi Utkarsh Porwal, and Andrew Trotman (Eds.).
  28. Jizhe Wang, Pipei Huang, Huan Zhao, Zhibo Zhang, Binqiang Zhao, and Dik Lun Lee. 2018. Billion-scale commodity embedding for e-commerce recommendation in alibaba. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. 839-848.
  29. Zhao Yan, Nan Duan, Peng Chen, Ming Zhou, Jianshe Zhou, and Zhoujun Li. 2017. Building Task-Oriented Dialogue Systems for Online Shopping. In Proceed- ings of the Thirty-First AAAI Conference on Artificial Intelligence (San Francisco, California, USA) (AAAI'17). AAAI Press, 4618-4625.
  30. Hang Yu and Lester Litchfield. 2020. Query Classification with Multi-objective Backoff Optimization. In Proceedings of the 43rd International ACM SIGIR Con- ference on Research and Development in Information Retrieval. 1925-1928.

Cited by

Sorted by citations

Authors
Papenmeier, A., Kern, D., Hienert, D., Sliwa, A., Aker, A., & Fuhr, N.
Affiliations
  • GESIS - Leibniz Institute for the Social Sciences (DE)
  • University of Duisburg-Essen (DE)
ORCID
Venue
Proceedings of the 2021 Conference on Human Information Interaction and Retrieval
Publisher
ACM
Published
2021-03-14
Pages
307-311
DOI
10.1145/3406522.3446043
OpenAlex
W3135158085
Cited by
8
References
1
Citations by year
2024: 4 · 2023: 2 · 2022: 2
FWCI
1.66
Type
conference paper
Language
en
Open access
Licence
Locations
  • Conference on Human Information Interaction and Retrieval (publishedVersion)
  • arXiv (Cornell University) (submittedVersion, cc-by-nc-sa)
  • arXiv (Cornell University)
Funders
Deutsche Forschungsgemeinschaft (388815326)
Topics
  • Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
Keywords
natural language queries, e-commerce search, product search, conversational search, vague terms, information needs, interactive information retrieval, voice assistants, chatbots

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review