CoChat
Sign Up
PreprintCited by 1Information Retrieval and Search BehaviorTopic ModelingNatural Language Processing Techniques

QueryNER: Segmentation of E-commerce Queries

Open access
Palen-Michel, Chester, Liang, Lizzie, Wu, Zhe, Lignos, Constantine·2024·arXiv
DOI 10.48550/arxiv.2405.09507arXiv 2405.09507
Open this paper in CoChat — free accountOpen PDF

Abstract

We present QueryNER, a manually-annotated dataset and accompanying model for e-commerce query segmentation. Prior work in sequence labeling for e-commerce has largely addressed aspect-value extraction which focuses on extracting portions of a product title or query for narrowly defined aspects. Our work instead focuses on the goal of dividing a query into meaningful chunks with broadly applicable types. We report baseline tagging results and conduct experiments comparing token and entity dropping for null and low recall query recovery. Challenging test sets are created using automatic transformations and show how simple data augmentation techniques can make the models more robust to noise. We make the QueryNER dataset publicly available.

AI snapshot

Written from the paper’s full text.

TL;DR

QueryNER introduces a manually annotated dataset and benchmark for segmenting e-commerce queries into 17 broad chunk types, using a subset of the Amazon Shopping Queries Dataset. Baseline BERT/XLM-R models reach about 60–61 F1, and experiments show entity dropping and data augmentation can aid null/low-recall recovery and robustness, though the task remains challenging.

Key findings

  • QueryNER uses a 17-type ontology and BIO labels; the final dataset has 7,841 train queries/17,505 entities/28,457 tokens, 871 dev queries/1,930 entities/3,124 tokens, and 933 test queries/2,317 entities/3,610 tokens. Average query length is 3.63 tokens and average entity span is 1.60 tokens; the most frequent types are core_product_type (8,310), modifier (3,367), and creator (2,217).
  • Inter-annotator agreement was lower on the public QueryNER test set than on an internal trial: Fleiss’ Kappa was 38.4 entity-level and 59.4 token-level for QueryNER, versus 65.6 entity-level and 77.2 token-level for the internal trial.
  • Baseline tagging results were BERT F1 60.56 ±0.4, XLM-R F1 60.10 ±0.5, and BERT with continued pre-training on the rest of the ESCI queries F1 61.29 ±0.3; the BERT vs. continued-pretraining difference had a Wilcoxon rank-sum p-value of 0.0008.
  • In null/low-recall recovery over 5,471 queries, dropping an entity while keeping core_product_type had a lower mean decrease in relevance than randomly dropping two tokens while maintaining similar recall gains; data augmentation improved transformed test sets, e.g. all transformations F1 rose from 29.04 to 67.77 and butterfinger from 32.19 to 54.73, but augmented training slightly worsened original-test performance (no augmentation 61.29 vs. all transformations 55.19).

Methods

  • Dataset creation used a subset of the Amazon Shopping Queries Dataset, a 17-type ontology, BIO format, and manual annotation; three annotators labeled the test data, one labeled train/dev, and test annotations were adjudicated. The authors began with 10,000 queries and removed some for language or profanity.
  • Baseline experiments trained sequence labeling models with BERT, XLM-R, and BERT with continued pre-training on the rest of the Amazon ESCI queries, using HuggingFace with batch size 16, 20 epochs, learning rate 5.0e-5, warmup ratio 0.1, 10 random seeds, and “conlleval” repair for invalid BIO sequences.
  • Additional experiments compared token dropping vs. entity dropping on 5,471 null/low-recall queries through an internal IR system, and tested five data augmentation transformations (shuffled, butterfingers, color, mention replacement, numeric, all transformations) by training on concatenated transformed and original data.

Limitations

  • The paper states ontology ambiguity: deciding between core_product_type and modifier, longer noun-noun compounds such as “dog cone collar,” span mismatches, and ambiguous tokens like “mazda” as creator vs. product name. It also states the dataset is English-only, was created for specific business use cases, comes from the Shopping Queries Dataset, and may not cover other subdivisions or categories; internal vs. public data also differed in entity-type distribution.
  • The paper reports low public inter-annotator agreement (Fleiss entity 38.4, token 59.4) and baseline F1 around 60–61, far below CoNLL NER F1 in the nineties, which the text makes evident as a sign of task difficulty and ambiguity.
  • The text makes evident that token vs. entity dropping experiments are not directly comparable because the average entity is 1.6 tokens while token-drop-2 removes two tokens, and that augmented training improves robustness to transformed test sets at the cost of roughly two F1 points on the original test set.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

DataCite · 10.48550/arxiv.2405.09507

Cites

0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

4 of 30 cited sources checked · 2 supported · 0 flagged · accuracy not yet measured

  • Findingown data

    Random dropping of two tokens does appear to have more cases where the gain in recall is substantial.

    “While random dropping of two tokens does appear to have more cases where the gain in recall is substantial, it is possible many of these query rewrites are lower in relevance and relaxing the constraints too much or had very low recall in the original query.”
    Quote in the paperNot a substantial claim
  • Findingown data

    There are very few cases with such a large relevance increase.

    “Note the right most peak in the distribution for the relevance experiment in Figure 4 is on a log scale and so there are very few that have such a large relevance increase.”
    Quote in the paperNot a substantial claim
  • Findingown data

    Overall the mean decrease in relevance over all the queries was lower for dropping an entity and keeping the core product type than for randomly dropping two tokens while maintaining similar gains in recall.

    “However, overall the mean decrease in relevance over all the queries was lower for dropping an entity and keeping the core product type than for randomly dropping two tokens while maintaining similar gains in recall.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Returning relevant results for null and low recall queries is an important challenge in e-commerce query understanding.
    • Rests on an assumption: Null and low recall queries return few or no results due to vocabulary mismatch or queries containing too many terms that over-constrain the retrieval process.
  • Findingown data

    In the data augmentation experiments on transformed test sets, with Butterfinger augmentation and transformed+original training, the model achieved precision 54.59 ±0.7, recall 54.88 ±0.5, and F1 54.73 ±0.6.

    “Table 7 : Data augmentation experiments on transformed test sets”
    • Also stated in: § Tag · plain
      “| Butterfinger | 30.86 ±0.6 | 33.65 ±0.5 32.19 ±0.5 54.59 ±0.7 | 54.88 ±0.5 54.73 ±0.6 | |”
    Quote in the paperNot a substantial claim
  • Findingown data

    Performance improves on the transformed test sets when training a model on both the transformed training data concatenated with the original training data.

    “As expected, performance improves on the transformed test sets when training a model on both the transformed training data concatenated with the original training data.”
    Quote in the paperSubstantial claim
  • May combine two claims. 25
  • Classified as probably not a substantial claim. 44
  • The quote only partly states this claim. 1

  • May combine two claims.

    Premisecitation

    Chunking is often framed as a sequence labeling task, and while there has been sequence labeling work in e-commerce, it has largely focused on aspect-value extraction.

    “Chunking is often framed as a sequence labeling task, and while there has been sequence labeling work in e-commerce, it has largely focused on aspect-value extraction.”
    Quote in the paperSubstantial claim1 of 2 sources checked
    • cites Joshi et al. 2015

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Papenmeier et al. 2021 source

      Section 3.2 (page unavailable) describes an e-commerce query annotation task conducted as sequence labelling, supporting the existence of such work. Neither that section nor §4.2 says that e-commerce sequence-labelling work has largely focused on aspect-value extraction; this paper labels key facts and vague words instead. (LLM, conf=0.86)

      Experimental — accuracy not yet measured

  • Classified as probably not a substantial claim.

    Premisecitation

    Aspect-value extraction identifies portions of a string of text (values) for more narrowly defined aspects like brand or color.

    “Aspect-value extraction identifies portions of a string of text (values) for more narrowly defined aspects like brand or color (Joshi et al. 2015;Papenmeier et al. 2021).”
    Quote in the paperSubstantial claim1 of 2 sources checked
    • cites Joshi et al. 2015

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Papenmeier et al. 2021 source

      Section 3.2 (page unavailable) describes labeling words or phrases in product queries as key facts, and Section 4.2 discusses identifying their attribute domains. This supports extracting text spans associated with product attributes, but the paper does not define aspect-value extraction or give brand and color as examples of extracted aspect-value pairs. (LLM, conf=0.82)

      Experimental — accuracy not yet measured

  • Classified as probably not a substantial claim.

    Premisecitation

    Further pre-training has been shown to increase scores when adapting a model to a specific domain or task.

    “Further pre-training has been shown to increase scores when adapting a model to a specific domain or task (Gururangan et al., 2020;Lee et al., 2020).”
    Quote in the paperSubstantial claim2 of 2 sources checked
    • cites Gururangan et al., 2020 source
    • cites Lee et al., 2020 source

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data

    The vast majority of entities are one or two tokens long, but there are smaller quantities that do have longer lengths.

    “Entity lengths are shown on a log scale to show that while the vast majority of entities are one or two tokens long, there are smaller quantities that do have longer lengths.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    The average of the lengths of all queries is 3.63 tokens.

    “The average of the lengths of all queries is 3.63 tokens, and the distribution of lengths of queries are shown in Figure 1.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given

    The internal data included more automotive parts and accessories than the Shopping Queries Dataset.

    “For example, the internal data included more automotive parts and accessories than the Shopping Queries Dataset.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    The next most frequent types are modifier and creator.

    “The next most frequent types are modifier and creator.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data

    The internal trial annotation was a smaller amount of data and consisted of only 400 queries with three annotators per query, while the subset of the Amazon Shopping Queries dataset that was annotated for QueryNER included 1,000 queries with three annotators per query for the test set.

    “The internal trial annotation also was a smaller amount of data and consisted of only 400 queries with three annotators per query, while the subset of the Amazon Shopping Queries dataset that was annotated for QueryNER included 1,000 queries with three annotators per query for the test set.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Challenging test sets are created using automatic transformations and show how simple data augmentation techniques can make the models more robust to noise.

    “Challenging test sets are created using automatic transformations and show how simple data augmentation techniques can make the models more robust to noise.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given

    QueryNER consists of an ontology of 17 types.

    “QueryNER consists of an ontology of 17 types.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingreasoning

    QueryNER has nearly all tokens included in some form of span, with the exception of a few special characters, some prepositions and conjunctions.

    “This difference in approach leads QueryNER to have nearly all tokens included in some form of span, with the exception of a few special characters, some prepositions and conjunctions.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Cohen's kappa values for the public dataset were more consistent across annotator pairs than the internal annotation where it appeared annotator 2 tended to conflict more with annotators 1 and 3.

    “Cohen's kappa values for the public dataset were more consistent across annotator pairs than the internal annotation where it appeared annotator 2 tended to conflict more with annotators 1 and 3.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The final QueryNER dataset contains close to 1,000 queries in the test set and over 7,000 queries in the training set.

    “The final QueryNER dataset contains close to 1,000 queries in the test set and over 7,000 queries in the training set.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    The average length of an entity span is 1.60 tokens.

    “The average length of an entity span is 1.60 tokens.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingreasoning

    core_product_type is the most frequent type.

    “Unsurprisingly, core_product_type is the most frequent type since most queries have a main product.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Agreement was relatively high in the initial internal annotation, but was lower when annotating the publicly released Amazon Shopping Queries Dataset.

    “Agreement was relatively high in the initial internal annotation, but was lower when annotating the publicly released Amazon Shopping Queries Dataset.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The difference between BERT and BERT with continued pretraining on the rest of the Amazon ESCI dataset's queries has a p-value of 0.0008 and is statistically significant despite being a fairly small difference in F1 score.

    “Using a Wilcoxon rank-sum test, the difference between BERT and BERT with continued pretraining on the rest of the Amazon ESCI dataset's queries has a p-value of 0.0008 and is statistically significant despite being a fairly small difference in F1 score.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    In the data augmentation experiments reported in Table 7, with Butterfinger augmentation and transformed+original training, the model achieved precision 58.93 ±0.5, recall 59.78 ±0.5, and F1 59.35 ±0.5.

    “Table 7 : Data augmentation experiments on transformed test sets”
    • Also stated in: § Tag · plain
      “| Butterfinger | 58.93 ±0.5 | 59.78 ±0.5 59.35 ±0.5 | | |”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given

    The dataset is only in English.

    “Another limitation is that the dataset is only in English, though we did a small internal experiment showing the promise of multilingual transfer to Spanish with similar performance as the English data.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The internal data had more years and dates and more frequently discussed the condition of an item, so the entity types time and condition were more prominent in the internally annotated data.

    “The internal data had more years and dates and more frequently discussed the condition of an item, so the entity types time and condition were more prominent in the internally annotated data.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data

    The QueryNER test split contains 933 queries, 2,317 entities, and 3,610 tokens.

    “Table 3 : Counts of queries, entities, and tokens in each of the QueryNER dataset splits”
    • Also stated in: § Tag · plain
      “| Test | 933 | 2,317 | 3,610 |”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    In the data augmentation experiments reported in Table 7, with Mention Replacement augmentation and transformed+original training, the model achieved precision 60.04 ±0.8, recall 59.50 ±0.7, and F1 59.77 ±0.7.

    “Table 7 : Data augmentation experiments on transformed test sets”
    • Also stated in: § Tag · plain
      “| Mention Replacement 60.04 ±0.8 | 59.50 ±0.7 59.77 ±0.7 | | |”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    In the data augmentation experiments reported in Table 7, with Numeric augmentation and transformed+original training, the model achieved precision 60.41 ±0.7, recall 59.35 ±0.7, and F1 59.88 ±0.7.

    “Table 7 : Data augmentation experiments on transformed test sets”
    • Also stated in: § Tag · plain
      “| Numeric | 60.41 ±0.7 | 59.35 ±0.7 59.88 ±0.7 | | |”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The ontology appeared to cover the public shopping queries dataset and was tested internally on multiple shopping categories.

    “While the ontology appeared to cover the public shopping queries dataset and was tested internally on multiple shopping categories, other subdivisions of entity types may be useful for other use cases or specific categories.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingreasoning

    It can be difficult for annotators to decide whether the correct chunking should be [dog cone] [collar], [dog][cone collar], or [dog cone collar].

    “It can be difficult for annotators to decide whether the correct chunking should be [dog cone] [collar], [dog][cone collar], or [dog cone collar].”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). The quote only partly states this claim.

    Findingown data

    In the data augmentation experiments reported in Table 7, with no augmentation and transformed+original training, the model achieved precision 61.78 ±0.4, recall 60.82 ±0.3, and F1 61.29 ±0.3.

    “Table 7 : Data augmentation experiments on transformed test sets”
    • Also stated in: § Tag · plain
      “| No Augmentation | 61.78 ±0.4 | 60.82 ±0.3 61.29 ±0.3 | | |”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingreasoning

    Many of the annotator disagreements come from slight span mismatches despite guidelines urging annotators to prefer longer more complete spans.

    “Many of the annotator disagreements come from slight span mismatches despite guidelines urging annotators to prefer longer more complete spans.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    The QueryNER development split contains 871 queries, 1,930 entities, and 3,124 tokens.

    “Table 3 : Counts of queries, entities, and tokens in each of the QueryNER dataset splits”
    • Also stated in: § Tag · plain
      “| Dev | 871 | 1,930 | 3,124 |”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The BERT baseline on QueryNER achieved precision 60.94 ±0.5, recall 60.17 ±0.4, and F1 60.56 ±0.4.

    “Table 5 . The baseline scores demonstrate the challenge of”
    • Also stated in: § Tag · plain
      “| BERT | 60.94 ±0.5 60.17 ±0.4 60.56 ±0.4 | |”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    In the data augmentation experiments reported in Table 7, with Color augmentation and transformed+original training, the model achieved precision 60.58 ±0.7, recall 59.05 ±0.6, and F1 59.80 ±0.6.

    “Table 7 : Data augmentation experiments on transformed test sets”
    • Also stated in: § Tag · plain
      “| Color | | 60.58 ±0.7 | 59.05 ±0.6 59.80 ±0.6 | |”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    A small internal experiment showed the promise of multilingual transfer to Spanish with similar performance as the English data.

    “Another limitation is that the dataset is only in English, though we did a small internal experiment showing the promise of multilingual transfer to Spanish with similar performance as the English data.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data

    The QueryNER training split contains 7,841 queries, 17,505 entities, and 28,457 tokens.

    “Table 3 : Counts of queries, entities, and tokens in each of the QueryNER dataset splits”
    • Also stated in: § Tag · plain
      “| Train | 7,841 | 17,505 28,457 | |”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    In the data augmentation experiments reported in Table 7, with Shuffled augmentation and transformed+original training, the model achieved precision 60.32 ±0.5, recall 59.97 ±0.6, and F1 60.14 ±0.5.

    “Table 7 : Data augmentation experiments on transformed test sets”
    • Also stated in: § Tag · plain
      “| Shuffled | 60.32 ±0.5 | 59.97 ±0.6 60.14 ±0.5 | | |”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingreasoning

    There are a number of ambiguous tokens that can be difficult to differentiate between types.

    “There are also a number of ambiguous tokens that can be difficult to differentiate between types.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    In the data augmentation experiments reported in Table 7, with All Transformations augmentation and transformed+original training, the model achieved precision 55.65 ±0.7, recall 54.74 ±0.7, and F1 55.19 ±0.7.

    “Table 7 : Data augmentation experiments on transformed test sets”
    • Also stated in: § Tag · plain
      “| All Transformations | 55.65 ±0.7 | 54.74 ±0.7 55.19 ±0.7 | | |”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The XLM-R baseline on QueryNER achieved precision 60.45 ±0.5, recall 59.75 ±0.5, and F1 60.10 ±0.5.

    “Table 5 . The baseline scores demonstrate the challenge of”
    • Also stated in: § Tag · plain
      “| XLM-R | 60.45 ±0.5 59.75 ±0.5 60.10 ±0.5 | |”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    There is notable gain in the challenge transformed test sets for the butterfinger transformation, shuffled transformation and all transformations applied at once, despite roughly two points of F1 degradation on the original test set.

    “It is notable, however, that despite this roughly two points of F1 degredation in performance on the original test set, there is notable gain in the challenge transformed test sets for the butterfinger transformation, shuffled transformation and all transformations applied at once.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Past work in NLP has treated grouping tokens as a shallow parsing or chunking task.

    “Past work in natural language processing (NLP) has treated grouping tokens as a shallow parsing or chunking task (Abney, 1992;Ramshaw and Marcus, 1995;Molina and Pla, 2002).”
    Quote in the paperNot a substantial claim0 of 3 sources checked
    • cites Abney, 1992

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Ramshaw and Marcus, 1995

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Molina and Pla, 2002

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Premisecitation

    Aspect-value extraction approaches tend to have either few aspect types with many tokens not included as part of a span, or they have large complex aspect ontologies with thousands of aspects.

    “Aspect-value extraction ap-proaches tend to have either few aspect types with many tokens not included as part of a span, or they have large complex aspect ontologies with thousands of aspects (Xu et al., 2019).”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Xu et al., 2019

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Classified as probably not a substantial claim.

    Premisecitation

    Much of the broader sequence labeling datasets that are publicly available to NLP researchers focus on the task of Named Entity Recognition (NER).

    “Much of the broader sequence labeling datasets that are publicly available to NLP researchers focus on the task of Named Entity Recognition (NER).”
    Quote in the paperSubstantial claim0 of 3 sources checked
    • cites Tjong Kim Sang 2002

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Tjong Kim Sang and De Meulder 2003

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Hovy et al. 2006 source

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    The test set was further adjudicated to resolve conflicts and review annotation.

    “The test set was further adjudicated to resolve conflicts and review annotation.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    One main difference in the guidelines given to annotators was to mark the fullest extent of a span possible.

    “One main difference in the guidelines given to annotators was to mark the fullest extent of a span possible.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Interpretationreasoning

    Many of the query rewrites from random token dropping are lower in relevance and relaxing the constraints too much or had very low recall in the original query.

    “While random dropping of two tokens does appear to have more cases where the gain in recall is substantial, it is possible many of these query rewrites are lower in relevance and relaxing the constraints too much or had very low recall in the original query.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Premisereasoning

    While aspect-value extraction does identify some natural groupings of tokens, the goal is often only to identify spans that are values for predefined aspects.

    “While aspect-value extraction does identify some natural groupings of tokens, the goal is often only to identify spans that are values for predefined aspects.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Interpretationnone given

    The tag types of QueryNER aim to be broadly applicable to queries for any product category.

    “Unlike the ecommerce task of aspect extraction, which tends to focus on fine-grained types that are often specific to particular categories of items, the tag types of QueryNER aim to be broadly applicable to queries for any product category.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Palen-Michel et al. (2021) highlighted some sequence labeling evaluation issues from invalid BIO label sequences.

    “Palen-Michel et al. (2021) highlighted some sequence labeling evaluation issues from invalid BIO label sequences.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Palen-Michel et al. (2021)

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    Additional quality checks were conducted which included flagging queries without a core product type or multiple core product types for further review.

    “Additional quality checks were conducted which included flagging queries without a core product type or multiple core product types for further review.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning

    A proportional gain of thousands of items may suggest that the query has been relaxed too much.

    “A proportional gain of thousands of items may suggest that the query has been relaxed too much, though we note that there are few of these cases.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Study design and conductown data

    Since the average number of tokens in an entity is 1.6 tokens and so differs from the random dropping of two tokens, these experiments are not directly comparable.

    “Since the average number of tokens in an entity is 1.6 tokens and so differs from the random dropping of two tokens, these experiments are not directly comparable.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S4 (based). May combine two claims.

    Interpretationnone given

    The type ontology is intended to be a small number of entity types and general purpose enough that it can be used for a broad range of e-commerce product categories.

    “The type ontology is intended to be a small number of entity types and general purpose enough that it can be used for a broad range of e-commerce product categories.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    All experiments are run using 10 different random seeds.

    “All experiments are run using 10 different random seeds.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    Three annotators were assigned to the test data in order to assess agreement and ensure quality.

    “Three annotators were assigned to the test data in order to assess agreement and ensure quality.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Interpretationreasoning

    Annotators do not necessarily need to become domain experts in the products involved in the annotation process or familiarize themselves with thousands of aspects.

    “Annotators also do not necessarily need to become domain experts in the products involved in the annotation process or familiarize themselves with thousands of aspects.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning

    By chunking a query, we can find the boundaries between spans of tokens and identify the purpose of each span.

    “By chunking a query, we can find the boundaries between spans of tokens and identify the purpose of each span.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning

    Chunking allows us to weight spans rather than tokens, drop spans as a recovery approach, and potentially better cluster similar chunks and link them to a knowledge graph.

    “This allows us to weight spans rather than tokens, drop spans as a recovery approach, and potentially better cluster similar chunks and link them to a knowledge graph.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Premisecitation

    A common approach to recover from null and low recall queries is to rewrite terms with similar words or to remove terms from the query to relax the constraints.

    “A common approach to recover from null and low recall queries is to rewrite terms with similar words or to remove terms from the query to relax the constraints.”
    Quote in the paperSubstantial claim0 of 3 sources checked
    • cites Bilotti, 2004

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Chen and Zhang, 2009

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Tan et al., 2017

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • Rests on an assumption: Returning relevant results for null and low recall queries is an important challenge in e-commerce query understanding.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Performance on the English CoNLL NER task has F1 scores reported in the nineties.

    “For comparison, performance on the English CoNLL NER task has F1 scores reported in the nineties (e.g.Chiu and Nichols 2016; Akbik et al. 2018).”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites Chiu and Nichols 2016

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Akbik et al. 2018

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Interpretationown data

    Using artificially augmented training data can help the model to be more robust to this type of noise, but at a slight cost of performance when measuring on the original test set.

    “We showed that using artificially augmented training data can help the model to be more robust to this type of noise, but at a slight cost of performance when measuring on the original test set.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S1 (asserted).

    Interpretationreasoning

    Since the core product and creator from the original dataset were preserved, perhaps these served as sufficient cues to make the dataset less challenging.

    “Since the core product and creator from the original dataset were preserved, perhaps these served as sufficient cues to make the dataset less challenging.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationown data

    Training on a combination of augmented data can help produce a robust segmentation model that performs well even under cases with spelling mistakes and free word order, but at the cost of a slight drop in performance on clean less noisy data.

    “These experiments demonstrate that if the goal is to produce a robust segmenta-tion model that performs well even under cases with spelling mistakes and free word order, training on a combination of augmented data can help, but at the cost of a slight drop in performance on clean less noisy data.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

    Interpretationreasoning

    Other subdivisions of entity types may be useful for other use cases or specific categories.

    “While the ontology appeared to cover the public shopping queries dataset and was tested internally on multiple shopping categories, other subdivisions of entity types may be useful for other use cases or specific categories.”
    Quote in the paperNot a substantial claim
61 of 61

  • The paper relies on this without support.

    Returning relevant results for null and low recall queries is an important challenge in e-commerce query understanding.

    “An important challenge in e-commerce query understanding is returning relevant results for null and low recall queries.”

  • The paper relies on this without support.

    Null and low recall queries return few or no results due to vocabulary mismatch or queries containing too many terms that over-constrain the retrieval process.

    “These queries return few or no results due to vocabulary mismatch or queries containing too many terms that over-constrain the retrieval process.”
2 of 2

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • Color augmentation does not improve transformed-test F1

    Minor

    The blanket claim of improved performance has an exception: on the color-transformed test set, F1 decreases from 61.08 to 60.98 with augmented training.

  • Overall query-length average does not match the split counts

    Minor

    The split counts total 35,191 tokens across 9,645 queries, giving an average of 3.65 tokens per query, not 3.63.

  • Entity-type counts exceed the split entity counts by two

    Minor

    The entity-type counts sum to 21,754, while the train, development, and test entity counts sum to 21,752.

3 of 3

Cites

The publisher hasn’t deposited this paper’s reference list.

Research works printed but not linked: 11

  1. Steven P Abney. 1992. Parsing by chunks. Principle-based parsing: Computation and Psy- cholinguistics, pages 257-278.
  2. Alan Akbik, Duncan Blythe, and Roland Vollgraf. 2018. Contextual string embeddings for se- quence labeling. In Proceedings of the 27th International Conference on Computational Lin- guistics, pages 1638-1649, Santa Fe, New Mex- ico, USA. Association for Computational Linguis- tics.
  3. Matthew William Bilotti. 2004. Query expansion techniques for question answering. Ph.D. thesis, Massachusetts Institute of Technology.
  4. Yan Chen and Yan-Qing Zhang. 2009. A query substitution-search result refinement ap- proach for long query web searches. In 2009 IEEE/WIC/ACM International Joint Conference on Web Intelligence and Intelligent Agent Tech- nology, volume 1, pages 245-251. IEEE. · DOI
  5. Jason P.C. Chiu and Eric Nichols. 2016. Named entity recognition with bidirectional LSTM-CNNs. Transactions of the Association for Computa- tional Linguistics, 4:357-370. · DOI
  6. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wen- zek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representa- tion learning at scale. In Proceedings of the 58th Annual Meeting of the Association for Computa- tional Linguistics, pages 8440-8451, Online. As- sociation for Computational Linguistics. · DOI · Paper page
  7. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language un- derstanding. In Proceedings of the 2019 Confer- ence of the North American Chapter of the As- sociation for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Lin- guistics. · DOI · Paper page
  8. Shahla Farzana, Qunzhi Zhou, and Petar Ristoski. 2023. Knowledge graph-enhanced neural query rewriting. In Companion Proceedings of the ACM Web Conference 2023, pages 911-919. · DOI
  9. Iftah Gamzu, Marina Haikin, and Nissim Halabi. 2020. Query rewriting for voice shopping null queries. In Proceedings of the 43rd International ACM SIGIR Conference on Research and De- velopment in Information Retrieval, pages 1369- 1378. · DOI
  10. Suchin Gururangan, Ana Marasović, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A. Smith. 2020. Don't stop pretraining: Adapt language models to domains and tasks. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8342-8360, Online. Associ- ation for Computational Linguistics. · DOI · Paper page
  11. Mahesh Joshi, Ethan Hart, Mirko Vogel, and Jean- David Ruvini. 2015. Distributed word repre- sentations improve NER for e-commerce. In Proceedings of the 1st Workshop on Vector Space Modeling for Natural Language Process- ing, pages 160-167, Denver, Colorado. Associ- ation for Computational Linguistics. · DOI

Cited by

Sorted by citations

Related

Authors
Palen-Michel, C., Liang, L., Wu, Z., & Lignos, C.
Affiliations
Brandeis University (US)
ORCID
Lignos, Constantine
Venue
arXiv
Publisher
arXiv
Published
2024
Pages
13455-13470
DOI
10.48550/arxiv.2405.09507
arXiv
2405.09507
OpenAlex
W7131903388
Cited by
1
References
0
Citations by year
2025: 1
FWCI
0.38
Type
conference paper
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • Language Resources and Evaluation Conference (LREC) (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Funders
Brandeis University
Topics
  • Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
Keywords
query segmentation, named entity recognition, query understanding

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-11-02.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-11-02.

Cited sources are checked by gpt-6-sol.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review