CoChat
Sign Up
Cited by 85Information Retrieval and Search BehaviorRecommender Systems and TechniquesWeb Data Mining and Analysis

Turning Clicks into Purchases: Revenue Optimization for Product Search in E-Commerce

Open access
Wu, Liang, Hu, Diane, Hong, Liangjie, Liu, Huan·2018·The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval
DOI 10.1145/3209978.3209993
Open this paper in CoChat — free accountOpen PDF

Abstract

In recent years, product search engines have emerged as a key factor for online businesses. According to a recent survey, over 55% of online customers begin their online shopping journey by searching on an E-Commerce (EC) website like Amazon as opposed to a generic web search engine like Google. Information retrieval research to date has been focused on optimizing search ranking algorithms for web documents while little attention has been paid to product search. There are several intrinsic differences between web search and product search that make the direct application of traditional search ranking algorithms to EC search platforms difficult. First, the success of web and product search is measured differently; one seeks to optimize for relevance while the other must optimize for both relevance and revenue. Second, when using real-world EC transaction data, there is no access to manually annotated labels. In this paper, we address these differences with a novel learning framework for EC product search called LETORIF (LEarning TO Rank with Implicit Feedback). In this framework, we utilize implicit user feedback signals (such as user clicks and purchases) and jointly model the different stages of the shopping journey to optimize for EC sales revenue. We conduct experiments on real-world EC transaction data and introduce a a new evaluation metric to estimate expected revenue after re-ranking. Experimental results show that LETORIF outperforms top competitors in improving purchase rates and total revenue earned.

AI snapshot

Written from the paper’s full text.

TL;DR

LETORIF is a learning-to-rank framework for e-commerce product search that jointly models user clicks (ranking) and purchases (classification) from implicit feedback to optimize expected Gross Merchandise Volume. On Etsy search logs, it achieves the best revenue-based NDCG and outperforms baselines on Rev@K for K > 1.

Key findings

  • LETORIF achieves the best Revenue NDCG@5 among all compared methods and the best Purchase NDCG@5; it is runner-up to MART on Rev@1 (4.58 vs. 4.62) but outperforms all baselines for 1 < K ≤ 10, reaching Rev@10 = 6.54.
  • Click-based methods generally outperform purchase-based methods on Click NDCG@5, with LambdaMART best and MART runner-up; purchase-based methods outperform click-based methods on Purchase NDCG except MART.
  • Incorporating price into the purchase model gives LETORIF a larger margin on Revenue NDCG, focusing optimization on items with greater estimated revenue.
  • In the Etsy data, nearly 70% of purchases occur below the first search-result position, motivating evaluation beyond top-1 revenue.

Methods

  • Data: 4 weeks of Etsy search logs filtered to queries with at least 1 purchase and over 100 impressions, yielding 334,931 sessions, 239,928 queries, 6,347,251 items, 270,239 buyers, 550,025 sellers, and 631,778 keywords; average items per session = 19.0 and per query = 26.5.
  • Features: 54 query–item features, split into 51 relevance features (TF/IDF/ICF variants, BM25, LMIR with DIR/JM/ABS smoothing) and 3 revenue features (price, price-category mean, deviation from category mean).
  • Model: LETORIF decomposes purchase probability into a click model and a purchase model; the click model optimizes a relaxed list-wise NDCG objective with sigmoid-approximated pairwise ranks, while the purchase model uses logistic regression with price weighting. Training alternates click and purchase model updates with early stopping.
  • Evaluation: five-fold cross-validation for click-, purchase-, and revenue-based NDCG@5; chronological 80/20 split for Rev@K (K = 1 to 10), compared against 12 baselines including RankNet, RankBoost, AdaRank, LambdaRank, ListNet, MART, LambdaMART, SVM, logistic regression, random forest, weighted purchase, and LMRM.

Limitations

  • Stated: the work focuses on implicit user feedback; future work could explore manual annotators or crowdsourcing workers to label the dataset, and personalization could further boost performance.
  • Stated: Rev@K assumes that an item actually purchased will also be purchased under the new ranking order if shown in the top K positions; the paper notes real users may click various results instead of a fixed number.
  • Evident from the text: experiments use search logs from a single e-commerce platform (Etsy) and a filtered query set requiring at least one purchase and over 100 impressions, so generalization to other platforms or unfiltered traffic is not established.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1145/3209978.3209993

Cites

48 of 48 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

0 of 4 cited sources checked · 0 supported · 0 flagged

  • Findingown data

    LMART achieves the best Click NDCG@5 among all methods.

    “LMART achieves the best Click NDCG@5 among all methods”
    Quote in the paperSubstantial claim
  • Findingown data

    LETORIF achieves the best Revenue NDCG among all methods.

    “LETORIF achieves the best Revenue NDCG among all methods”
    Quote in the paperSubstantial claim
  • Findingreasoning

    Click-based methods generally perform better than purchase-based methods.

    “Click-based methods generally perform better than purchase-based methods”
    Quote in the paperSubstantial claim
  • Findingown data

    By linearly integrating purchase information, WT outperforms most relevance methods except MART.

    “By linearly integrating purchase information, WT outperforms most relevance methods except MART.”
    Quote in the paperNot a substantial claim
  • Findingown data

    When K = 1, MART achieves the best actual revenue.

    “When K = 1, MART achieves the best actual revenue.”
    Quote in the paperNot a substantial claim
  • May combine two claims. 19
  • Classified as probably not a substantial claim. 22
  • The quote only partly states this claim. 1

  • May combine two claims.

    Findingreasoning

    LMART and RM get relatively high NDCG@5 on the training dataset but are outperformed by the runner-up method MART.

    “Though LMART and RM get relatively high NDCG@5 on the training dataset, they are outperformed by the runner-up method MART”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    LETORIF outperforms top competitors in improving purchase rates and total revenue earned.

    “Experimental results show that LETORIF outperforms top competitors in improving purchase rates and total revenue earned.”
    • Also stated in: § 1 INTRODUCTION · boosted
      “Experiments on real-world data show the superiority of LETORIF over other competitive baselines, such as Random Forest and LambdaMART.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    The choice of regression method did not significantly impact performance in the authors' experience.

    “Note that other regression methods can also be used here, though in our experience the choice did not signiicantly impact performance.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingreasoning

    LMRM achieves high training and validation NDCG while being outperformed by LETORIF on test.

    “Similar to LMART, LMRM achieves a high training and validation NDCG while outperformed by LETORIF”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    MART achieves the best score among click optimization methods and RM achieves the best score among purchase optimization methods.

    “MART and RM achieve the best score among click and purchase optimization methods, respectively.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    MART is the runner-up method and RNet also achieves relatively high NDCG@5.

    “MART is the runner-up method and the other click-based method RNet also achieves relatively high NDCG@5.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    As an individual model, LETORIF performs best under most settings and is the runner-up with respect to Rev@1.

    “As an individual model LETORIF performs best under most settings and is the runner-up w.r .t .Rev@1.”
    Quote in the paperSubstantial claim

  • The quote only partly states this claim.

    Findingown data

    LETORIF is the runner-up method when K = 1.

    “LETORIF is the runner-up method in the setting.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Premisenone given

    Product search in EC consists of two stages: clicking on a product from the search result page and deciding whether to purchase.

    “Product search in EC consists of two stages: (1) clicking on a product from the search result page and (2) deciding whether to purchase.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisereasoning

    There is no standard way of obtaining ground-truth information in EC product search because manual annotation is nearly infeasible.

    “However, there is no standard way of obtaining ground-truth information in the context of EC as employing manual annotation is nearly infeasible: hundreds of products may be similarly relevant to a query.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    The study randomly split labeled search sessions into a training and a validation set.

    “We input the labeled search sessions S which is randomly split into a training and a validation set in line 2.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Interpretationreasoning

    Incorporating the price of each item substantially increases the output of the purchase model when the items are more expensive.

    “Since cheaper items are usually purchased more frequently, incorporating the price of each item substantially increases the output of the purchase model when the items are more expensive.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: Cheaper items are usually purchased more frequently.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisereasoning

    List-wise approaches jointly model the entire group of items.

    “In order to tackle the challenge, list-wise approaches jointly model the entire group of items.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning

    The first stage of an EC product search session is a ranking problem, while the second stage is a binary classification problem.

    “As such, the irst stage is a ranking problem, where listings should be presented to the consumer in the order of relevance, while the second stage is a binary classiication problem, which models the consumer's decision to purchase or not.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Premisenone given

    The proposed method is the first piece of work that aims to discover the optimal ranking algorithm for optimizing revenue for an EC search platform.

    “To the best of our knowledge, this is the irst piece of work that aims to discover the optimal ranking algorithm for optimizing revenue for an EC search platform.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning

    Product search is more a ranking problem and the interdependency between products exists.

    “which indicates that product search is more a ranking problem and the interdependency between products exists.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    NDCG is a common evaluation metric for ranked lists.

    “A common evaluation metric for ranked lists is Normalized Discounted Cumulative Gain (NDCG) [13].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [13]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Classified as probably not a substantial claim.

    Study design and conductnone given

    The study chronologically partitioned search sessions into 80/20 training/test splits.

    “We chronologically partition search sessions into 80/20 splits, where the irst split is used for training and the second for test.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductown data

    The study used early stopping based on validation loss to control early termination of optimization.

    “We set the maximum number of iterations to be Max it er , and also use a function EarlyStop() to control early termination of the optimization process, taking the loss on the validation set as input.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Premisenone given

    In real-world EC transaction data, there is no access to manually annotated labels.

    “Second, when using real-world EC transaction data, there is no access to manually annotated labels.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisenone given

    Point-wise ranking approaches assume items are independent and identically distributed and do not consider interdependency among documents.

    “However, point-wise approaches assume items to be independent and identically distributed (i.i.d.) and the interdependency among web documents is not considered.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Premisenone given

    Traditional evaluation metrics are mainly designed for measuring relevance between search queries and web documents.

    “Traditional evaluation metrics are mainly designed for measuring relevance between search queries and web documents.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisecitation

    Pair-wise LETOR methods directly model the relative order of items, making positions of items indirectly visible to the learning algorithm.

    “Instead, pair-wise LETOR methods directly model the relative order of items.”
    • Also stated in: § 3.1 Optimizing Click · plain
      “As such, positions of items are indirectly visible to a pair-wise LETOR method [2].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [2]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Interpretationreasoning

    The new rank relaxation formulation is smooth and convex and can be conveniently optimized.

    “In addition, the new formulation is smooth and convex and can be conveniently optimized.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S4 (based). May combine two claims.

    Premisenone given

    A purchased product must have been clicked, while a clicked product is not necessarily purchased.

    “Note that, a purchased product must have been clicked, while a clicked one is not necessarily to be purchased, so we omitted Ψ here.”
    • Also stated in: § 3.3 Proposed Method for Product Search · plain
      “We focus on a typical search session, so being clicked is a prerequisite for an item to be purchased Pr (Φ = 1|Ψ = 0, i, q, u) = 0.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

    Premisenone given

    CTR reflects the utility of a listing and can be used for annotation and constructing a benchmark dataset.

    “CT R(i |q) relects the łutilityž of a listing and can be used for annotation and constructing a benchmark dataset.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductnone given

    The study used five-fold cross-validation, shuffling all data instances and randomly splitting them into five folds.

    “In order to perform a ive-fold crossvalidation, we shule all data instances and randomly split them into ive folds.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based). May combine two claims.

    Premisenone given

    Web search success is measured by relevance, while product search must optimize for both relevance and revenue.

    “First, the success of web and product search is measured diferently; one seeks to optimize for relevance while the other must optimize for both relevance and revenue.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

    Interpretationreasoning

    Rev@K provides a clearer perspective to understand how ranking methods impact actual revenue in a real-world setting.

    “which provides a clearer perspective to understand how the ranking methods impact the actual revenue in a real-world setting.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    WT is the state-of-the-art method for optimizing revenue in computational advertising.

    “WT is the state-of-the-art method for optimizing revenue in computational advertising.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [41]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductnone given

    Five-fold cross-validation was adopted and a holdout set was used as a validation set.

    “Five-fold cross-validation has been adopted and a holdout set is used as a validation set.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    All model parameters of baseline methods were tuned on the holdout validation set.

    “All model parameters of baseline methods (e.g., the number of trees, tree depths) are tuned on the holdout validation set.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Premisenone given

    Revenue optimization of product search and advertising are intrinsically different because the price of a product is the cost a user has to pay while the price of an ad is the cost that an advertiser bids to pay.

    “revenue optimization of product search and advertising are intrinsically diferent since price of a product is the cost a user has to pay while price of an ad is the cost that an advertiser bids to pay.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning

    The superiority of LETORIF over the other two joint baselines WT and LMRM reveals that linearly integrating purchases with a higher weight or completely dealing with the two problems independently cannot achieve the optimal performance.

    “The superiority of LETORIF over the other two joint baselines WT and LMRM reveals that linearly integrating purchases with a higher weight or completely dealing with the two problems independently cannot achieve the optimal performance.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S4 (based). May combine two claims.

    Premisenone given

    Online metrics are concentrated on web search and ignore revenue, which is the key measure to evaluate success of a product search engine.

    “However, these metrics are concentrated on web search, and ignoring revenue, which is the key measure to evaluate success of a product search engine”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning

    The problem becomes a point-wise task if only the top ranked item brings about revenue.

    “The problem here becomes a point-wise task if only the top ranked item brings about revenue.”
    Quote in the paperNot a substantial claim
36 of 36

  • The paper relies on this without support.

    Cheaper items are usually purchased more frequently.

    “Since cheaper items are usually purchased more frequently, incorporating the price of each item substantially increases the output of the purchase model when the items are more expensive.”

  • The authors state this as an assumption.

    An item that has actually been purchased will also be purchased under the new ranking order if it is shown in the top K positions.

    “We assume that an item that has been actually purchased, will also be purchased under the new ranking order if it is shown in the top K positions.”

  • The authors state this as an assumption.

    Top K is assumed to measure the ranking quality for optimizing revenue.

    “We make the assumption of top K to measure the ranking quality for optimizing revenue.”

  • The authors state this as an assumption.

    The display position is omitted in the purchase model; purchase is conditionally independent of position given click.

    “The display position (p) is omitted in the purchase model, namely (Φ ⊥ ⊥ P) | Ψ, (9 ) where the position is only considered in the search result page.”

  • The paper relies on this without support.

    Implicit user feedback is prevalently available on EC platforms and can be used to reveal the quality of a search result.

    “Meanwhile, implicit user feedback is prevalently available on EC platforms and can be used to reveal the łqualityž of a search result.”

  • The paper relies on this without support.

    The purchase of an item is independent.

    “though the purchase of an item is independent”

  • The paper relies on this without support.

    User satisfaction for a search session cannot be directly obtained in product search.

    “the user satisfaction for a search session cannot be directly obtained”

  • The paper relies on this without support.

    Ignoring purchase behaviors makes EC search sessions identical to a web document dataset.

    “Ignoring purchase behaviors makes the EC search sessions identical to a web document dataset.”

  • The paper relies on this without support.

    There is no significant difference between different evaluation metrics.

    “In fact, there is no signiicant diference between diferent metrics.”

  • The paper relies on this without support.

    LMART is the best performing method in benchmark studies of web search.

    “consistent with prior observations on benchmark studies of web search that LMART is the best performing method.”
10 of 10

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • Smoothed rank reverses the stated score ordering

    Major

    Equation (11) gives an item a smaller rank when its predicted click score exceeds its competitors' scores. The purported smoothed version in Equation (15) uses a sigmoid of the score difference in the opposite direction, giving that item a larger rank contribution instead.

  • Purchase-model gradient does not match its loss function

    Minor

    The stated loss includes a positive squared-norm penalty, whose gradient is +2 times the parameter. Equation (20) gives −2 times the parameter; its data term also omits the feature and logistic factor needed to differentiate Equation (19).

  • Revenue equation counts click probability twice

    Minor

    Equation (8) defines the unconditional purchase probability as click probability multiplied by purchase probability conditional on a click. Equation (21) instead multiplies click probability by that same unconditional purchase probability, which would count the click probability twice.

  • Rank formula assigns the top item rank zero

    Minor

    The text says rank 1 is the top position, but Equation (11) counts competitors whose scores are at least as high. An item with a uniquely highest score therefore receives rank 0.

  • RNet revenue falls when the top-K cutoff increases

    Minor

    Table 6 reports RNet Rev@6 as 5.23 and Rev@7 as 5.21. Under the stated Rev@K rule, expanding the cutoff retains every purchase counted at K=6 and can only add purchases, so revenue cannot decrease.

5 of 5

Cites

Flagged first, then by citations

Research works printed but not linked: 4

  1. Freund Yoav, 2003, An eicient boosting algorithm for combining preferences, Journal of machine learning research 4, 933
  2. 10.5555/646943.712093 · DOI
  3. Kullback Solomon, 1951, On information and suiciency. The annals of mathematical statistics 22, 1
  4. Liaw Andy, 2002, Matthew Wiener, et al

Cited by

Sorted by citations

Related

Authors
Wu, L., Hu, D., Hong, L., & Liu, H.
Affiliations
  • Arizona State University (US)
  • Etsy (United States) (US)
ORCID
Venue
The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval
Publisher
ACM
Published
2018-06-27
Pages
365-374
DOI
10.1145/3209978.3209993
OpenAlex
W2798283910
Semantic Scholar
305f4f38d74aeb96f7c4b3822f0dba110e8f2bc3
Cited by
85
References
47
Citations by year
2026: 3 · 2025: 8 · 2024: 10 · 2023: 8 · 2022: 9 · 2021: 13 · 2020: 16 · 2019: 17 · 2018: 1
FWCI
9.22
Type
conference paper
Language
en
Open access
Licence
www.acm.org/publications/policies/copyright_policy#Background (vor)
Locations
International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) (publishedVersion)
Funders
  • National Science Foundation (1614576)
  • Office of Naval Research (N00014-16-1-2257)
Topics
  • Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
  • Physical Sciences › Computer Science › Information Systems › Recommender Systems and Techniques
  • Physical Sciences › Computer Science › Information Systems › Web Data Mining and Analysis
Keywords
product search, implicit feedback, learning to rank, click-through data, revenue optimization, re-ranking

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-11-02.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-11-02.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review