CoChat
Sign Up
PreprintCited by 21Information Retrieval and Search BehaviorRecommender Systems and TechniquesTopic Modeling

Shopping Queries Dataset: A Large-Scale ESCI Benchmark for Improving Product Search

Open access
Reddy, Chandan K., Màrquez, Lluís, Valero, Fran, Rao, Nikhil, Zaragoza, Hugo, Bandyopadhyay, Sambaran, Biswas, Arnab, Xing, Anlu +1 more·2022·arXiv
DOI 10.48550/arxiv.2206.06588arXiv 2206.06588
Open this paper in CoChat — free accountOpen PDF

Abstract

Improving the quality of search results can significantly enhance users experience and engagement with search engines. In spite of several recent advancements in the fields of machine learning and data mining, correctly classifying items for a particular user search query has been a long-standing challenge, which still has a large room for improvement. This paper introduces the "Shopping Queries Dataset", a large dataset of difficult Amazon search queries and results, publicly released with the aim of fostering research in improving the quality of search results. The dataset contains around 130 thousand unique queries and 2.6 million manually labeled (query,product) relevance judgements. The dataset is multilingual with queries in English, Japanese, and Spanish. The Shopping Queries Dataset is being used in one of the KDDCup'22 challenges. In this paper, we describe the dataset and present three evaluation tasks along with baseline results: (i) ranking the results list, (ii) classifying product results into relevance categories, and (iii) identifying substitute products for a given query. We anticipate that this data will become the gold standard for future research in the topic of product search.

AI snapshot

Written from the paper’s full text.

TL;DR

The paper introduces the Shopping Queries Dataset, a large multilingual Amazon product-search benchmark with about 130,000 unique queries and 2.6 million manually labeled (query, product) ESCI relevance judgments. It defines three KDD Cup ’22 tasks—ranking, multiclass product classification, and substitute identification—and reports baseline results.

Key findings

  • The large dataset version contains 130,652 unique queries and 2,621,738 judgments across English, Spanish, and Japanese; the small ranking version contains 48,300 unique queries and 1,118,170 judgments in Table 1.
  • ESCI labels are imbalanced: in the large version, Exact = 65.20%, Substitute = 21.91%, Complement = 2.89%, Irrelevant = 10.00%; in the small version, Exact = 43.72%, Substitute = 34.33%, Complement = 5.13%, Irrelevant = 16.82%.
  • Baseline public-test results: Task 1 nDCG is 0.852 for the fine-tuned Cross-Encoder/MPNet approach versus 0.563 for BM25; Task 2 micro-F1 is 0.656 overall (English 0.685, Spanish 0.580, Japanese 0.595); Task 3 micro-F1 is 0.780 overall (English 0.795, Spanish 0.757, Japanese 0.737).
  • Human annotation agreement is 91% overall, rising to >96% for a binary Exact vs. Not-Exact distinction.

Methods

  • The dataset was built from challenging Amazon search queries selected via behavioral, parse-pattern, negation, price-pattern, NLQEC, and other sampling strategies, then manually annotated with E/S/C/I labels using at least three annotations per pair and majority vote.
  • It provides query-product pairs with product title, description, bullet points, color, brand, locale, and ESCI label; splits are stratified by query at 70% train, 15% public test, and 15% private test.
  • Baselines include BM25, a fine-tuned MS MARCO Cross-Encoder for English and multilingual MPNet models for Spanish/Japanese for ranking, and a frozen BERT multilingual MLP classifier for multiclass classification and substitute identification.

Limitations

  • The paper states that human labels are not perfect and can be noisy; estimated annotation accuracy is 91%, with most discrepancies being Irrelevant cases judged as valid Substitutes and some annotator bias toward being less strict on Substitute.
  • The paper states that query selection is biased by the production baseline models used to find difficult queries, and that private-test information is omitted because the competition was ongoing at the time of writing.
  • Evident from the text: the BM25 comparison is not fair because it used default configuration and performed poorly on Japanese due to non-Japanese-specific preprocessing; the paper also notes large room for improvement and substantial English vs. Spanish/Japanese classification gaps.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

DataCite · 10.48550/arxiv.2206.06588

Cites

0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

4 of 10 cited sources checked · 2 supported · 0 flagged · accuracy not yet measured

Verify all 1 references — free account
  • Interpretationnone given
    § 2.1 Query Selection Process · p. 2

    The sampling approach biased by the selection of baseline models leads to results that are more interesting (harder for many models) than those obtained by using sampling methods directly tied to the properties of a particular model.

    “While this approach is clearly biased by the selection of baseline models, in our experience, it leads to results that are more interesting (i.e., harder for many models) than those obtained by using sampling methods directly tied to the properties of a particular model (such as those obtained by active learning or adversarial approaches used to improve model training such as [3, 15]).”
    Quote in the paperSubstantial claim1 of 2 sources checked
    • cites [3] [3] Mustafa Bilgic and Paul N. Bennett. 2012. Active Query Selection for Learning Rankers. In ACM SIGIR Conference on Research and Development in Information Retrieval . http://www.cs.iit.edu/~ml/pdfs/bilgic-sigir12.pdf - source
    • cites [15] [15] Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2020. Winogrande: An adversarial winograd schema challenge at scale. In Proceedings of the AAAI Conference on Artificial Intelligence , Vol. 34. 8732-8740. - source

      The citation is attached to the example of an adversarial approach, not to the citing authors’ comparison with their own sampling method. Section 3 (p. 3) identifies AFLITE as an adversarial filtering method, but describes its purpose as reducing bias in a benchmark, not improving model training. The source also does not compare AFLITE with the citing authors’ sampling approach. (LLM, conf=0.83)

      Experimental — accuracy not yet measured
  • Findingown data
    § 1 INTRODUCTION · p. 1

    Each query-product pair is accompanied by additional information from the Amazon catalog, including product title, product description, and additional product related bullet points.

    “Each query-product pair is accompanied by additional information from the Amazon catalog, including: product title, product description, and additional product related bullet points.”
    Quote in the paperNot a substantial claim
  • Findingcitation
    § 1 INTRODUCTION · p. 1

    For each query, the dataset provides a list of up to 40 results together with their ESCI relevance judgements (Exact, Substitute, Complement, or Irrelevant).

    “For each query, the dataset provides a list of up to 40 results, together with their ESCI relevance judgements (Exact, Substitute, Complement, or Irrelevant) indicating the relevance of the product to the query [11].”
    • Also stated in: § 3.1 Task 1: Query-Product Ranking · p. 3 · plain
      “The maximum number of products per query is 40, and at least one is guaranteed to be non irrelevant (either Exact, Substitute, or Complement).”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [11] [11] Julian McAuley, Rahul Pandey, and Jure Leskovec. 2015. Inferring networks of substitutable and complementary products. In Proceedings of the 21th ACM SIGKDD international conference on knowledge discovery and data mining . 785794. - source
  • Findingcitation
    § ABSTRACT · p. 1

    The Shopping Queries Dataset is multilingual, containing queries in English, Japanese, and Spanish.

    “The dataset is multilingual with queries in English, Japanese, and Spanish.”
    • Also stated in: § 1 INTRODUCTION · p. 1 · plain
      “The Shopping Queries Dataset is multilingual, as it contains queries in English, Japanese, and Spanish [1].”
    • Also stated in: § 2 SHOPPING QUERIES DATASET · p. 2 · plain
      “The data is multilingual, and it includes queries from English, Japanese, and Spanish languages.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [1] [1] Aman Ahuja, Nikhil Rao, Sumeet Katariya, Karthik Subbian, and Chandan K Reddy. 2020. Language-agnostic representation learning for product search on 2. e-commerce platforms. In Proceedings of the 13th International Conference on Web Search and Data Mining . 7-15. - source
  • Findingnone given
    § 1 INTRODUCTION · p. 1

    The collection is derived from real customers searching for real products online.

    “First, it is derived from real customers searching for real products online.”
    Quote in the paperSubstantial claim
  • Classified as probably not a substantial claim. 14
  • May combine two claims. 15

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § 1 INTRODUCTION · p. 1

    Improving the relevance of search results can significantly improve the customer experience and their engagement with search.

    “Improving the relevance of search results can significantly improve the customer experience and their engagement with search [4].”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites [4] [4] Nurendra Choudhary, Nikhil Rao, Sumeet Katariya, Karthik Subbian, and Chandan K. Reddy. 2022. ANTHEM: Attentive Hyperbolic Entity Model for Product Search. In Proceedings of the Fifteenth ACM International Conference on Web Search and Data Mining (Virtual Event, AZ, USA) (WSDM '22) . Association for Computing Machinery, New York, NY, USA, 161-171. https://doi.org/10.1145/ 3488560.3498456 - source

  • May combine two claims.

    Findingown data
    § 1 INTRODUCTION · p. 1

    The dataset provides both breadth (a large number of queries, in three languages) and depth (approximately 20 results per query), unlike other existing large document retrieval collections which tend to provide either breadth or depth but not both.

    “Second, it provides both breadth (a large number of queries, in three languages) and depth ( ≈ 20 results per query), unlike other existing large document retrieval collections which tend to provide either breadth or depth but not both (e.g. MSMarco [2], TRECDL [6], NLQEC [13]).”
    Quote in the paperSubstantial claim2 of 3 sources checked
    • cites [2] [2] Payal Bajaj, Daniel Campos, Nick Craswell, Li Deng, Jianfeng Gao, Xiaodong Liu, Rangan Majumder, Andrew McNamara, Bhaskar Mitra, Tri Nguyen, Mir Rosenberg, Xia Song, Alina Stoica, Saurabh Tiwary, and Tong Wang. 2016. MS MARCO:AHumanGenerated MAchine Reading COmprehension Dataset. https: //doi.org/10.48550/ARXIV.1611.09268 - source
    • cites [6] [6] Nick Craswell, Bhaskar Mitra, Emine Yilmaz, Daniel Campos, Ellen M Voorhees, and Ian Soboroff. 2021. TREC deep learning track: Reusable test collections in the large data regime. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval . 2369-2375. - source
    • cites [13] [13] Andrea Papenmeier, Dagmar Kern, Daniel Hienert, Alfred Sliwa, Ahmet Aker, and Norbert Fuhr. 2021. Dataset of Natural Language Queries for E-Commerce. Proceedings of the 2021 Conference on Human Information Interaction and Retrieval (2021). - source

      Split into 2 atoms — 1 confirmed, 1 partially supported, 0 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.

      Experimental — accuracy not yet measured

  • May combine two claims.

    Findingown data
    § ABSTRACT · p. 1

    The Shopping Queries Dataset (large version) contains approximately 130,000 unique queries and 2.6 million manually labeled (query,product) relevance judgements.

    “The dataset contains around 130 thousand unique queries and 2.6 million manually labeled (query,product) relevance judgements.”
    • Also stated in: § 2.4 Splits and Statistics · p. 3 · plain
      “The larger version of the dataset contains 130,652 unique queries and 2,621,738 judgements, corresponding each to a (query-result) judgement.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingreasoning
    § 1 INTRODUCTION · p. 1

    The additional product information in the dataset is public, as it is displayed at the Amazon website when searching for those products.

    “This information is public, as it is displayed at the Amazon website when searching for those products.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data
    § 2.4 Splits and Statistics · p. 3

    The average number of results per query in English is slightly smaller than in Spanish and Japanese, which are comparable.

    “On the contrary, the average number of results per query in English is slightly smaller than in Spanish and Japanese, which are comparable.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § 2.4 Splits and Statistics · p. 3

    The paper provides two different versions of the dataset, one for Task 1 (query-product ranking) and another which is a superset of the first for Tasks 2 and 3 (multiclass product classification and product substitute identification).

    “We provide two different versions of the dataset, one for Task1 (query-product ranking) and another which is a superset of the first for Tasks 2 and 3 (multiclass product classification, and product substitute identification).”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § 4.2 Results · p. 4

    In Task 3, the results are still favorable to English but with smaller differences, half the size compared to the difference seen in Task 2.

    “In Task 3, the results are still favorable to English but with smaller differences, half the size compared to the difference seen in Task 2.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data
    § 2.4 Splits and Statistics · p. 3

    The smaller version of the dataset contains 48,300 unique queries and 1,118,117 rows.

    “The smaller version of the dataset contains 48,300 unique queries and 1,118,117 rows.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data
    § 2.4 Splits and Statistics · p. 3

    Across languages, the proportion of English queries (61.8% and 74.5% for the small and large versions, respectively) is significantly larger than the proportion of Spanish (16.6% and 11.6%) and Japanese (21.5% and 13.9%) queries.

    “Across languages, we can see that the proportion of English queries (61.8% and 74.5% for the small and large versions, respectively ) is significantly larger than the proportion of Spanish (16.6% and 11.6%) and Japanese (21.5% and 13.9%) queries.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § 4.2 Results · p. 4

    The neural approach obtains nDCG scores in the interval (0.840, 0.857), which are comparable across languages.

    “The neural approach obtains nDCG scores in the interval (0.840, 0.857), which are comparable across languages.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given
    § 2.4 Splits and Statistics · p. 3

    The smaller version is a subset of the larger version where the simpler queries (in terms of NDCG) are filtered out.

    “The smaller version is a subset of the larger version where the simpler queries (in terms of NDCG) are filtered out.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingnone given
    § 3.1 Task 1: Query-Product Ranking · p. 3

    The maximum number of products per query is 40, and at least one product is guaranteed to be non-irrelevant (Exact, Substitute, or Complement).

    “The maximum number of products per query is 40, and at least one is guaranteed to be non irrelevant (either Exact, Substitute, or Complement).”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given
    § 5 CONCLUSION · p. 5

    The three evaluation tasks explained in the paper are the ones proposed in the KDDCup'22 challenge.

    “Then, we explain three evaluation tasks, which are the ones proposed in the KDDCup'22 challenge, and we provide some initial results from various baselines for reference.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data
    § 2.4 Splits and Statistics · p. 3

    The class labels are imbalanced, with label E being the most frequent and label C being the least.

    “The class labels are imbalanced, with label E being the most frequent, and label C being the least.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § 1 INTRODUCTION · p. 2

    The dataset provides descriptions of retrieval objects with categorical and textual metadata, as well as multiple levels of representation from a short title to a long description of the product.

    “Fifth, we also provide descriptions of the retrieval objects which have categorical and textual metadata, as well as multiple levels of representation (from a short title to a long description of the product).”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given
    § 1 INTRODUCTION · p. 2

    Products in the dataset are linked to the online Amazon catalog.

    “Finally, products are linked to the online Amazon catalog.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § 4.2 Results · p. 4

    For the other two classification tasks, the BERT-based MLP classifier obtains results that are clearly better for English than for Spanish and Japanese (e.g., for Task 2, F1 for English and Spanish is 0.685 vs. 0.580).

    “For the other two classification tasks, the BERT-based MLP classifier obtains results that are clearly better for English than for Spanish and Japanese (e.g., for Task 2, compare the F1 for English and Spanish, 0.685 vs. 0.580).”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Findingnone given
    § 5 CONCLUSION · p. 5

    The Shopping Queries Dataset is a large-scale benchmark to improve state-of-the-art algorithms for e-commerce product search.

    “In this paper, we introduce the Shopping Queries Dataset, a largescale benchmark to improve the state-of-the-art algorithms for ecommerce product search.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationown data
    § 2.4 Splits and Statistics · p. 3

    The larger version is an 'easier' dataset than the smaller version, as the proportion of Exact matches is higher (65.2% vs 43.7%) and the number of results per query is smaller (20.1 vs 23.2).

    “Finally, we can observe that the larger version is an 'easier' dataset than the smaller version, as the proportion of Exact matches is higher (65.2% vs 43.7%) and the number of results per query is smaller (20.1 vs 23.2).”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Study design and conductown data
    § 2.2 Annotation of Relevance Judgements · p. 2

    A minimum of three annotations were collected for each query-result pair, and an automatic aggregation mechanism selected the majority vote as the gold label.

    “A minimum of three annotations were collected for each pair, and an automatic aggregation mechanism selected the majority vote as the gold label.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Study design and conductreasoning
    § 2.2 Annotation of Relevance Judgements · p. 2

    The resulting labels are not perfect and can be noisy because human annotation is involved in the process.

    “Since there is human annotation involved in the process, the resulting labels are not perfect and can be noisy.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Study design and conductreasoning
    § 4.2 Results · p. 4

    The comparison between the neural approach and Terrie-BM25 is not fair because the default configuration for Terrie-BM25 was used, which obtains poor results on Japanese due to non-Japanese specific preprocessing, thus significantly penalizing the averaged metric across the overall data.

    “It should be noted that this is not a fair comparison since we used the default configuration for the Terrie-BM25 approach, which obtains poor results on Japanese (due to non-Japanese specific preprocessing), thus significantly penalizing the averaged metric across the overall data.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationreasoning
    § 2.1 Query Selection Process · p. 2

    A randomly sampled query set would not be of great interest to the research community.

    “For this reason, a randomly sampled query set would not be of great interest to the research community.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: Most frequent shopping queries are easy to solve with standard state-of-the-art search engine techniques and result in near-perfect results.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Interpretationown data
    § 3.1 Task 1: Query-Product Ranking · p. 4

    There is a corner case where nDCG is not well defined when all results are Irrelevant, but this is not possible in the dataset since all queries have at least one non-irrelevant result.

    “Note that there is a corner case where nDCG is not well defined, i.e., when all results are Irrelevant. This is not possible in our case since all queries have at least one non irrelevant result.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § 3.1 Task 1: Query-Product Ranking · p. 4

    nDCG is a commonly used relevance metric in the literature.

    “This is a commonly used relevance metric in the literature.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [7] [7] Kalervo Järvelin and Jaana Kekäläinen. 2002. Cumulated gain-based evaluation of IR techniques. ACMTransactions on Information Systems 20(4) (2002), 422-446. - source
25 of 25

  • The paper relies on this without support.

    Most frequent shopping queries are easy to solve with standard state-of-the-art search engine techniques and result in near-perfect results.

    “Most frequent shopping queries are easy to solve with standard state-of-the-art search engine techniques and result in near-perfect results.”

  • The paper relies on this without support.

    The notion of binary relevance limits the customer experience; classifying each product as relevant or not typically yields results that have a detrimental effect on user experience.

    “Furthermore, the notion of binary relevance limits the customer experience. Specifically, classifying each product shown in response to a user query as being relevant or not typically yields results that have a detrimental effect on user experience.”

  • The paper relies on this without support.

    Extremely high accuracy in ranking is needed when developing machine learning models for online shopping applications, especially for mobile and voice search applications where even a small number of irrelevant items can significantly hurt the user experience.

    “When developing machine learning models for online shopping applications, extremely high accuracy in ranking is needed. This is even more stringent requirement when deploying search in mobile and voice search applications, where even a small number of irrelevant items can significantly hurt the user experience.”

  • The paper relies on this without support.

    The presence of noisy information in the results, the difficulty of understanding the query intent, and the diversity of the items available are some of the reasons that contribute to the complexity of correctly classifying items for a shopping query.

    “The presence of noisy information in the results, the difficulty of understanding the query intent, and the diversity of the items available are some of the reasons that contribute to the complexity of this problem.”
4 of 4

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Existing e-commerce datasets target retrieval relevance and use short, keyword-style queries, which do not resemble realistic interactions with shopping agents.

    “Existing e-commerce datasets target retrieval relevance and use short, keyword-style queries (Papenmeier et al., 2021; Chen et al., 2022; Reddy et al., 2022), which do not resemble realistic interactions with shopping agents.”

    From E-GEO: A Testbed for Generative Engine Optimization in E-Commerce (2025) · cites this paper as Reddy et al., 2022

  • WANDS and ESCI average ∼3-3.5 words in a single sentence and Papenmeier et al. (2021)'s queries average ∼33 words, while E-GEO queries average ∼59 words.

    “Quantitatively, WANDS (Chen et al., 2022) and ESCI (Reddy et al., 2022) average ∼ 3-3.5 words in a single sentence and Papenmeier et al. (2021)'s queries average ∼ 33 words , while E-GEO queries average ∼ 59 words (Table 2, which also lists representative example queries).”

    From E-GEO: A Testbed for Generative Engine Optimization in E-Commerce (2025) · cites this paper as Reddy et al., 2022

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • Small dataset judgement count differs from Table 1

    Minor

    The text reports 1,118,117 rows for the small dataset, while Table 1 reports 1,118,170 total judgements-a difference of 53.

  • BM25 nDCG differs from Table 4

    Minor

    The results discussion gives the BM25 nDCG as 0.551, but Table 4 reports 0.563 for the overall public test.

2 of 2

Cites

The publisher hasn’t deposited this paper’s reference list.

Research works printed but not linked: 4

  1. [8][8] Jacob Devlin Ming-Wei Chang Kenton and Lee Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of NAACL-HLT . 4171-4186. -
  2. [10][10] Craig Macdonald, Richard McCreadie, Rodrygo LT Santos, and Iadh Ounis. 2012. From puppy to maturity: Experiences in developing Terrier. Proc. of OSIR at SIGIR (2012), 60-63. -
  3. [12][12] Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. 2016. MS MARCO: A human generated machine reading comprehension dataset. In CoCo@ NIPS . -
  4. [17][17] C. J. Van Rijsbergen. 1979. Information Retrieval (2 ed.). Butterworth-Heinemann.

Cited by

Sorted by citations

Related

Authors
Reddy, C. K., Màrquez, L., Valero, F., Rao, N., Zaragoza, H., Bandyopadhyay, S., Biswas, A., Xing, A., & Subbian, K.
Affiliations
Amazon (United States) (US)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2022
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2206.06588
arXiv
2206.06588
OpenAlex
W4282961889
Semantic Scholar
46b259403b91be9643b1b689f1354e64f1da1879
Cited by
17
References
0
Citations by year
2026: 2 · 2025: 3 · 2024: 6 · 2023: 6
Type
posted content
Language
en
Open access
Licence
creativecommons.org/licenses/by-nc-sa/4.0/legalcode
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
  • Physical Sciences › Computer Science › Information Systems › Recommender Systems and Techniques
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
Keywords
product search, search result relevance, product substitution, search engine ranking

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-11-02.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-11-02.

Cited sources are checked by gpt-6-sol.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review