CoChat
Sign Up
Cited by 127Information Retrieval and Search BehaviorTopic ModelingAdvanced Image and Video Retrieval Techniques

Semantic Product Search

Open access
Nigam, Priyanka, Song, Yiwei, Mohan, Vijai, Lakshman, Vihan, Ding, Weitian (Allen), Shingavi, Ankit, Teo, Choon Hui, Gu, Hao +1 more·2019·Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining
DOI 10.1145/3292500.3330759
Open this paper in CoChat — free accountOpen PDF

Abstract

We study the problem of semantic matching in product search, that is, given a customer query, retrieve all semantically related products from the catalog. Pure lexical matching via an inverted index falls short in this respect due to several factors: a) lack of understanding of hypernyms, synonyms, and antonyms, b) fragility to morphological variants (e.g. "woman" vs. "women"), and c) sensitivity to spelling errors. To address these issues, we train a deep learning model for semantic matching using customer behavior data. Much of the recent work on large-scale semantic search using deep learning focuses on ranking for web search. In contrast, semantic matching for product search presents several novel challenges, which we elucidate in this paper. We address these challenges by a) developing a new loss function that has an inbuilt threshold to differentiate between random negative examples, impressed but not purchased examples, and positive examples (purchased items), b) using average pooling in conjunction with n-grams to capture short-range linguistic patterns, c) using hashing to handle out of vocabulary tokens, and d) using a model parallel training architecture to scale across 8 GPUs. We present compelling offline results that demonstrate at least 4.7% improvement in [email protected] and 14.5% improvement in mean average precision (MAP) over baseline state-of-the-art semantic search methods using the same tokenization method. Moreover, we present results and discuss learnings from online A/B tests which demonstrate the efficacy of our method.

AI snapshot

Written from the paper’s full text.

TL;DR

The paper develops a deep semantic matching model for product search trained on customer behavior, combining shared query/product embeddings, average pooling, n-gram/character-trigram tokenization, OOV hashing, and a 3-part hinge loss. It reports at least 4.7% Recall@100 and 14.5% MAP improvement over state-of-the-art semantic search baselines offline, with online A/B tests in three categories showing statistically significant KPI increases.

Key findings

  • The 3-part L2 hinge loss gave the best offline matching results in Table 1: Recall 0.735, MAP 0.664, matching NDCG 0.791, matching MRR 0.591, ranking NDCG 0.772, ranking MRR 0.973, outperforming 2-part hinge, BCE, MSE, and MAE variants.
  • Average pooling performed similarly to or slightly better than LSTM/GRU with less training time; the paper reports <0.5% difference in MAP and Recall@100 relative to recurrent approaches (Table 2).
  • The best tokenization was Unigrams+Bigrams+Char Trigrams+OOV, with Recall 0.794, MAP 0.745, matching NDCG 0.810, matching MRR 0.659, ranking NDCG 0.794, ranking MRR 0.980 (Table 3); OOV hashing improved matching, and larger OOV bins improved matching while ranking peaked at lower bin sizes.
  • Against baselines with unigrams and 3-part L2 loss, the model reached Recall 0.735/MAP 0.664 versus DSSM 0.702/0.614, Match Pyramid 0.562/0.450, and ARC-II 0.465/0.348; random embeddings reached 0.207/0.107. Online A/B tests across toys and games, kitchen, and pets showed statistically significant increases in conversion rate, revenue, and other KPIs.

Methods

  • Siamese factorized neural model with shared query/product embedding layer, average pooling, batch normalization, and cosine similarity; tokenization combines word unigrams, word n-grams, character trigrams, and hashing for OOV tokens. The 3-part hinge loss uses thresholds ε+=0.9, ε-=0.2, and ε0=0.55.
  • Training used 11 months of search logs and 1 month for evaluation; 54 billion sampled query-product training pairs were preprocessed to 650 million rows by grouping over time and weighting counts. For each query, products are purchased, impressed-but-not-purchased, or random, with a target ratio of 6 impressed and 7 random products per purchase.
  • Offline evaluation sampled 20K queries and a 1M-product sub-corpus, measuring Recall@100, MAP, NDCG, and MRR; training used embedding dimension 256, Xavier initialization, batch size 8192, ADAM (α=0.001, β1=0.9, β2=0.999, ε=10^-8), and model-parallel training across 8 GPUs.

Limitations

  • The paper states that online precision required guard rails through additional heuristics and ranking models to filter irrelevant products, and future work aims to improve precision and eliminate those heuristics; initial self-attention and positional encodings did not improve precision.
  • The paper states it focuses on matching and excludes latent factor models, and only presents factorized-model experiments; interaction models were noted as memory/computation expensive for online inference.
  • Evident from the text: offline evaluation uses a sampled 20K-query and 1M-product sub-corpus rather than the full catalog, and online A/B tests are reported only for three categories without effect sizes or confidence intervals.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1145/3292500.3330759

Cites

31 of 31 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

0 of 4 cited sources checked · 0 supported · 0 flagged

  • Findingown data

    Large variability in the accuracy and availability of structured data across products led to 5% lower recall than simply concatenating the attributes.

    “However, large variability in the accuracy and availability of structured data across products led to 5% lower recall than simply concatenating the attributes.”
    Quote in the paperSubstantial claim
  • Findingreasoning

    Including character trigrams improves performance further.

    “Additionally including character trigrams improves the performance further as character trigrams provide generalization and robustness to spelling errors.”
    Quote in the paperSubstantial claim
  • Findingown data

    Ranking performance peaks at lower OOV bin sizes.

    “We see that matching performance improves as the bin size increases, although ranking performance peaks at lower bin sizes.”
    Quote in the paperNot a substantial claim
  • Findingnone given

    Hashing unseen tokens improves the precision across different tokenization strategies.

    “Furthermore, we showed that hashing unseen tokens improves the precision across different tokenization strategies.”
    Quote in the paperSubstantial claim
  • Findingown data

    ARC-II and Match Pyramid baselines only achieve a sub-50% Recall@100 and MAP on the product matching dataset.

    “only achieve a sub-50% Recall@100 and MAP on our product matching dataset, as shown with the ARC-II and Match Pyramid baselines in Table 5.”
    Quote in the paperSubstantial claim
  • May combine two claims. 21
  • Classified as probably not a substantial claim. 19

  • May combine two claims.

    Findingown data

    During initial experiments with a bin size of 10,000, hashing collisions incorrectly promoted irrelevant products for queries, led to overfitting, and did not improve offline metrics.

    “During our initial experiments with a bin size of 10,000, we noticed that hashing collisions incorrectly promoted irrelevant products for queries, led to overfitting, and did not improve offline metrics.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given

    Preprocessing training data into tokens and using mmap decreases training time by 3X.

    “We also decrease the training time by 3X by preprocessing the training data into tokens and using mmap to store the tokens.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    A factorized model architecture achieves comparable performance while only requiring constant memory per product.

    “While we frequently evaluate our hypotheses on interaction matrix-based methods, we find that a factorized model architecture achieves comparable performance while only requiring constant memory per product.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingreasoning

    The 3-part hinge loss outperforms the 2-part hinge loss in matching metrics in all experiments, although the two loss functions have similar ranking performance.

    “The 3-part hinge loss outperforms the 2-part hinge loss in matching metrics in all experiments although the two loss functions have similar ranking performance.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Unlike web search, both query and product information in product search tend to be shorter, without long-range dependencies.

    “We reconciled this departure from state-of-the-art solutions for Question Answering and other NLP tasks through an analysis that showed that, unlike web search, both query and product information tend to be shorter, without long-range dependencies.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingnone given

    These models are also much harder to tune and tend to overfit.

    “These models are also much harder to tune and tend to overfit.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

    Findingnone given

    Unigram tokenization is used in Table 1 and Table 2, as the relative ordering of results does not change with other more sophisticated tokenizations.

    “Unigram tokenization is used in Table 1 and Table 2, as the relative ordering of results does not change with other more sophisticated tokenizations.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    More complex models, like Match Pyramid and ARC-II, had significantly lower matching and ranking performance while taking significantly longer to train and evaluate.

    “More complex models, like Match Pyramid and ARC-II, had significantly lower matching and ranking performance while taking significantly longer to train and evaluate.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given

    Previous experiments using other methods, including Glove and word2vec, to initialize the embeddings yielded poorer results than end-to-end training.

    “Previous experiments using other methods, including Glove and word2vec, to initialize the embeddings yielded poorer results than end-to-end training.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingreasoning

    Averaging token embeddings performs similar to or slightly better than recurrent units (LSTM, GRU) with significantly less training time.

    “Averaging performs similar to or slightly better than recurrent units with significantly less training time.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Sharing the embedding layer in a Siamese network works well, capturing local word-level matching even before training these networks.

    “Intuitively, sharing the embedding layer in a Siamese network works well, capturing local word-level matching even before training these networks.Our experiments in Table 7 confirm this intuition.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    ARC-II and Match Pyramid ranking performance is similar or lower when using the 3-part loss.

    “ARC-II and Match Pyramid ranking performance is similar or lower when using the 3-part loss.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The best model understands synonyms, intents, spelling errors and overall robustness, as shown by example queries and products retrieved in Table 6.

    “In Table 6, we show example queries and products retrieved to highlight the efficacy of our best model to understand synonyms, intents, spelling errors and overall robustness.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Using only character trigrams has competitive ranking but not matching performance compared to unigrams.

    “It is worth noting that using only character trigrams, which was an essential component of DSSM [11], has competitive ranking but not matching performance compared to unigrams.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Findingnone given

    Queries typically have fewer words than the product content.

    “Queries typically have fewer words than the product content.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given

    One challenge we faced with our semantic search solution was weeding out irrelevant results to meet the precision bar for production search quality.

    “One challenge we faced with our semantic search solution was weeding out irrelevant results to meet the precision bar for production search quality.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisereasoning

    Stemming algorithms are often imperfect and lead to information loss and errors.

    “However, stemming algorithms are often imperfect and lead to information loss and errors.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Premisenone given

    Lexical matching is sensitive to spelling errors.

    “Sensitivity to spelling errors.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Premisenone given

    Pure lexical matching via an inverted index fails to understand hypernyms, synonyms, and antonyms.

    “Pure lexical matching via an inverted index falls short in this respect due to several factors: a) lack of understanding of hypernyms, synonyms, and antonyms”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Premisenone given

    Lexical matching is fragile to morphological variants such as woman versus women.

    “Fragility to morphological variants (e.g.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    According to some estimates of web search logs, 10-15% of queries are misspelled.

    “According to some estimates of web search logs [3,4], 10-15% of queries are misspelled.”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites [3]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites [4]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S4 (based).

    Premisenone given

    Pointwise loss functions, such as MSE or MAE, require an additional step post-training to identify the threshold.

    “Pointwise loss functions, such as mean squared error (MSE) or mean absolute error (MAE), require an additional step post-training to identify the threshold.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning

    Character trigrams provide generalization and robustness to spelling errors.

    “Additionally including character trigrams improves the performance further as character trigrams provide generalization and robustness to spelling errors.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Premisereasoning

    Pairwise loss functions do not provide guarantees on the magnitude of scores, only on relative ordering, and thus do not work well in practice with threshold-based pruning.

    “Pairwise loss functions do not provide guarantees on the magnitude of scores (only on relative ordering) and thus do not work well in practice with threshold-based pruning.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Premisenone given

    Latent factor models improve ranking metrics due to their ability to memorize associations between the query and the product.

    “Although latent factor models improve ranking metrics due to their ability to memorize associations between the query and the product, we exclude it from this paper as we focus on the matching task.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationnone given

    Recurrent methods are more expressive but introduce specialization between the query and title.

    “Furthermore, recurrent methods are more expressive but introduce specialization between the query and title.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Premisenone given

    Interaction methods require memory and computation proportional to the number of words in the document and hence are prohibitively expensive for online inference.

    “Nevertheless, these interaction methods require memory and computation proportional to the number of words in the document and hence are prohibitively expensive for online inference.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Premisenone given

    Character trigrams have the ability to capture similarity of model parts and sizes.

    “Another advantage in our setting is the ability to capture similarity of model parts and sizes.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationown data

    Sharing the embedding layer helps local word-level matching and generalization to unseen tokens when using OOV bins.

    “As we described previously, sharing the embedding layer helps local word-level matching and generalization to unseen tokens when using OOV bins.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationnone given

    Average pooling requires far less computation, reducing training time and inference latency.

    “Average pooling also requires far less computation, reducing training time and inference latency.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Premisenone given

    Models based on conversion rates or click-through-rates may incorrectly favor accessories over the main product.

    “Models based on conversion rates or click-through-rates may incorrectly favor accessories (like a phone cover) over the main product (like a cell phone).”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Premisenone given

    Products have multiple attributes, like title, brand, and color, that are material to the matching process.

    “Products have multiple attributes, like title, brand, and color, that are material to the matching process.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning

    The model can better differentiate between positives and random negatives, improving Recall and MAP.

    “The model can better differentiate between positives and random negatives, improving Recall and MAP.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Premisenone given

    Shoppers may maintain multiple intents during a single search session.

    “This is further complicated by shoppers maintaining multiple intents during a single search session: a customer may be looking for a specific television model while also looking for accessories for this item at the lowest price and browsing additional products to qualify for free shipping.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based). May combine two claims.

    Premisenone given

    Matching should differentiate purchased and impressed products from random ones, while ranking should differentiate purchased products from impressed ones.

    “Intuitively, matching should differentiate purchased and impressed products from random ones; ranking should differentiate purchased products from impressed ones.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Premisereasoning

    Customer behavior logs contain semantic information.

    “Intuitively, there is reason to believe that customer behavior logs contain semantic information; customers who are intent on purchasing a product circumvent the limitations of lexical matching by query reformulation or by deeper exploration of the search results.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Premisenone given

    Character trigrams are robust to typos and handle compound words naturally.

    “Character trigrams are robust to typos (”
    Quote in the paperSubstantial claim
37 of 37

  • The paper relies on this without support.

    Product search queries tend to be shorter than web search queries and positive signals (purchases) are sparser than clicks.

    “Product search is different from web search as the queries tend to be shorter and the positive signals (purchases) are sparser than clicks.”

  • The paper relies on this without support.

    Product search queries do not contain stop words and typically require every query word (or its synonym) to be present in the product.

    “Additionally, product search queries do not contain stop words and typically require every query word (or its synonym) to be present in the product.”

  • The paper relies on this without support.

    The number of tokens is small in comparison with the token embeddings or the embedding matrix.

    “Sending the input to all GPUs tends to be inexpensive as the number of tokens is small in comparison with the token embeddings or the embedding matrix.”
3 of 3

Cites

Flagged first, then by citations

Research works printed but not linked: 4

  1. 10.5555/645757.666141 · DOI
  2. Dumais Susan T, AAAI spring symposium on cross-language text and speech retrieval
  3. Glorot Xavier, 2010, In Proceedings of the International Conference on Artificial Intelligence and Statistics (AISTATS'10)
  4. 10.5555/1394399 · DOI

Cited by

Sorted by citations

Related

Authors
Nigam, P., Song, Y., Mohan, V., Lakshman, V., Ding, W. (., Shingavi, A., Teo, C. H., Gu, H., & Yin, B.
Affiliations
  • Amazon (United States) (US)
  • Carnegie Mellon University (US)
ORCID
Venue
Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining
Publisher
ACM
Published
2019-07-25
Pages
2876-2885
DOI
10.1145/3292500.3330759
OpenAlex
W2964369530
Cited by
127
References
23
Citations by year
2026: 10 · 2025: 15 · 2024: 40 · 2023: 21 · 2022: 15 · 2021: 18 · 2020: 7 · 2019: 1
FWCI
8.16
Type
conference paper
Language
en
Open access
Licence
Locations
  • ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) (publishedVersion, cc-by-nc-nd)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Image and Video Retrieval Techniques
Keywords
semantic matching, e-commerce search, loss function design, average pooling, n-grams, hashing, A/B testing, customer behavior data, morphological variants, spelling errors

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-11-02.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-11-02.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review