CoChat
Sign Up
Cited by 33Information Retrieval and Search BehaviorRecommender Systems and TechniquesTopic Modeling

Learning a Product Relevance Model from Click-Through Data in E-Commerce

Open access
Yao, Shaowei, Tan, Jiwei, Chen, Xi, Yang, Keping, Xiao, Rong, Deng, Hongbo, Wan, Xiaojun·2021·Proceedings of the Web Conference 2021
DOI 10.1145/3442381.3450129
Open this paper in CoChat — free account

Abstract

The search engine plays a fundamental role in online e-commerce systems, to help users find the products they want from the massive product collections. Relevance is an essential requirement for e-commerce search, since showing products that do not match search query intent will degrade user experience. With the existence of vocabulary gap between user language of queries and seller language of products, measuring semantic relevance is necessary and neural networks are engaged to address this task. However, semantic relevance is different from click-through rate prediction in that no direct training signal is available. Most previous attempts learn relevance models from user click-through data that are cheap and abundant. Unfortunately, click behavior is noisy and misleading, which is affected by not only relevance but also factors including price, image and attractive titles. Therefore, it is challenging but valuable to learn relevance models from click-through data. In this paper, we propose a new relevance learning framework that concentrates on how to train a relevance model from the weak supervision of click-through data. Different from previous efforts that treat samples as either relevant or irrelevant, we construct more fine-grained samples for training. We propose a novel way to consider samples of different relevance confidence, and come up with a new training objective to learn a robust relevance model with desirable score distribution. The proposed model is evaluated on offline annotated data and online A/B testing, and it achieves both promising performance and high computational efficiency. The model has already been deployed online, serving the search traffic of Taobao for over a year.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1145/3442381.3450129

Cites

3 of 3 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • The ranking of mentioned products is often critical to consumer purchasing decisions.

    “We specifically consider the domain of consumer products, in which the ranking of mentioned products is often critical to consumer purchasing decisions (Yao et al., 2021).”

    From Ranking Manipulation for Conversational Search Engines (2024) · cites this paper as Yao et al., 2021

    The Introduction (p. 1) links e-commerce search to customer engagement and conversion. Section 3.1 (p. 4) says that higher-ranked search results receive more clicks. These passages support the importance of product order in online shopping, but they do not establish that the ranking of mentioned products is often critical to purchasing decisions. (LLM, conf=0.82)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 25

  1. [1][1] David Carmel, Elad Haramaty, Arnon Lazerson, Liane Lewin-Eytan, and Yoelle Maarek. 2020. Why Do People Buy Seemingly Irrelevant Items in Voice Product Search?: On the Relation between Product Relevance and Customer Satisfaction in eCommerce. In WSDM '20: The Thirteenth ACM International Conference on Web Search and Data Mining, Houston, TX, USA, February 3-7, 2020 . ACM, 79-87. https://doi.org/10.1145/3336191.3371780 - · DOI
  2. [2][2] Zhuyun Dai, Chenyan Xiong, Jamie Callan, and Zhiyuan Liu. 2018. Convolutional Neural Networks for Soft-Matching N-Grams in Ad-Hoc Search. In Proceedings of the Eleventh ACM International Conference on Web Search and Data Mining (Marina Del Rey, CA, USA) (WSDM '18) . Association for Computing Machinery, New York, NY, USA, 126-134. https://doi.org/10.1145/3159652.3159659 - · DOI
  3. [3][3] Jesse Davis and Mark Goadrich. 2006. The Relationship between PrecisionRecall and ROC Curves. In Proceedings of the 23rd International Conference on Machine Learning (Pittsburgh, Pennsylvania, USA) (ICML '06) . Association for Computing Machinery, New York, NY, USA, 233-240. https://doi.org/10.1145/ 1143844.1143874 -
  4. [4][4] Mostafa Dehghani, Hamed Zamani, Aliaksei Severyn, Jaap Kamps, and W. Bruce Croft. 2017. Neural Ranking Models with Weak Supervision. In Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval (Shinjuku, Tokyo, Japan) (SIGIR '17) . Association for Computing Machinery, New York, NY, USA, 65-74. https://doi.org/10.1145/ 3077136.3080832 -
  5. [6][6] Jiafeng Guo, Yixing Fan, Qingyao Ai, and W. Bruce Croft. 2016. A Deep Relevance Matching Model for Ad-Hoc Retrieval. In Proceedings of the 25th ACM International on Conference on Information and Knowledge Management (Indianapolis, Indiana, USA) (CIKM '16) . Association for Computing Machinery, New York, NY, USA, 55-64. https://doi.org/10.1145/2983323.2983769 - · DOI
  6. [7][7] Baotian Hu, Zhengdong Lu, Hang Li, and Qingcai Chen. 2014. Convolutional Neural Network Architectures for Matching Natural Language Sentences. In Proceedings of the 27th International Conference on Neural Information Processing Systems - Volume 2 (Montreal, Canada) (NIPS'14) . MIT Press, Cambridge, MA, USA, 2042-2050. -
  7. [8][8] Po-Sen Huang, Xiaodong He, Jianfeng Gao, Li Deng, Alex Acero, and Larry P. Heck. 2013. Learning deep structured semantic models for web search using clickthrough data. In 22nd ACM International Conference on Information and Knowledge Management, CIKM'13, San Francisco, CA, USA, October 27 - November 1, 2013 . ACM, 2333-2338. https://doi.org/10.1145/2505515.2505665 - · DOI
  8. [9][9] Yunjiang Jiang, Yue Shang, Rui Li, Wen-Yun Yang, Guoyu Tang, Chaoyi Ma, Yun Xiao, and Eric Zhao. 2019. A Unified Neural Network Approach to E-Commerce Relevance Learning. In Proceedings of the 1st International Workshop on Deep Learning Practice for High-Dimensional Sparse Data (Anchorage, Alaska) (DLPKDD '19) . Association for Computing Machinery, New York, NY, USA, Article 10, 7 pages. https://doi.org/10.1145/3326937.3341259 - · DOI
  9. [10][10] Thorsten Joachims, Adith Swaminathan, and Tobias Schnabel. 2017. Unbiased Learning-to-Rank with Biased Feedback. In Proceedings of the Tenth ACM International Conference on Web Search and Data Mining (Cambridge, United Kingdom) (WSDM'17) . Association for Computing Machinery, New York, NY, USA, 781-789. https://doi.org/10.1145/3018661.3018699 - · DOI · Paper page
  10. [11][11] Bhaskar Mitra and Nick Craswell. 2017. Neural models for information retrieval. arXiv preprint arXiv:1705.01509 (2017). - · arXiv
  11. [14][14] Rodrigo Nogueira, Wei Yang, Kyunghyun Cho, and Jimmy Lin. 2019. Multi-Stage Document Ranking with BERT. arXiv preprint arXiv:1910.14424 (2019). - · arXiv
  12. [15][15] Hamid Palangi, Li Deng, Yelong Shen, Jianfeng Gao, Xiaodong He, Jianshu Chen, Xinying Song, and R Ward. 2014. Semantic modelling with long-short-term memory for information retrieval. arXiv preprint arXiv:1412.6629 (2014). - · arXiv
  13. [16][16] Hamid Palangi, Li Deng, Yelong Shen, Jianfeng Gao, Xiaodong He, Jianshu Chen, Xinying Song, and Rabab Ward. 2016. Deep Sentence Embedding Using Long Short-Term Memory Networks: Analysis and Application to Information Retrieval. IEEE/ACM Trans. Audio, Speech and Lang. Proc. 24, 4 (April 2016), 694-707. https://doi.org/10.1109/TASLP.2016.2520371 - · DOI
  14. [17][17] Liang Pang, Yanyan Lan, Jiafeng Guo, Jun Xu, Shengxian Wan, and Xueqi Cheng. 2016. Text Matching as Image Recognition. In Proceedings of the Thirtieth AAAI 18. Conference on Artificial Intelligence (Phoenix, Arizona) (AAAI'16) . AAAI Press, 2793-2799. -
  15. [18][18] Liang Pang, Yanyan Lan, Jiafeng Guo, Jun Xu, Jingfang Xu, and Xueqi Cheng. 2017. DeepRank: A New Deep Architecture for Relevance Ranking in Information Retrieval. In Proceedings of the 2017 ACM on Conference on Information and Knowledge Management (Singapore, Singapore) (CIKM '17) . Association for Computing Machinery, New York, NY, USA, 257-266. https: //doi.org/10.1145/3132847.3132914 - · DOI
  16. [19][19] Ankur Parikh, Oscar Täckström, Dipanjan Das, and Jakob Uszkoreit. 2016. A Decomposable Attention Model for Natural Language Inference. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing . Association for Computational Linguistics, Austin, Texas, 2249-2255. https: //doi.org/10.18653/v1/D16-1244 - · DOI
  17. [20][20] Nils Reimers and Iryna Gurevych. 2019. Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) . Association for Computational Linguistics, Hong Kong, China, 3982-3992. https://doi.org/10.18653/v1/D19-1410 - · DOI · Paper page
  18. [21][21] Stephen Robertson, S. Walker, S. Jones, M. M. Hancock-Beaulieu, and M. Gatford. 1995. Okapi at TREC-3. In Overview of the Third Text REtrieval Conference (TREC3) . Gaithersburg, MD: NIST, 109-126. -
  19. [22][22] Yelong Shen, Xiaodong He, Jianfeng Gao, Li Deng, and Grégoire Mesnil. 2014. A Latent Semantic Model with Convolutional-Pooling Structure for Information Retrieval. In Proceedings of the 23rd ACM International Conference on Conference on Information and Knowledge Management (Shanghai, China) (CIKM '14) . Association for Computing Machinery, New York, NY, USA, 101-110. https://doi.org/10.1145/2661829.2661935 - · DOI
  20. [23][23] Shengxian Wan, Yanyan Lan, Jun Xu, Jiafeng Guo, Liang Pang, and Xueqi Cheng. 2016. Match-SRNN: Modeling the Recursive Matching Structure with Spatial RNN. In Proceedings of the Twenty-Fifth International Joint Conference on Artificial Intelligence (New York, New York, USA) (IJCAI'16) . AAAI Press, 2922-2928. -
  21. [24][24] Rong Xiao, Jianhui Ji, Baoliang Cui, Haihong Tang, Wenwu Ou, Yanghua Xiao, Jiwei Tan, and Xuan Ju. 2019. Weakly Supervised Co-Training of Query Rewriting And Semantic Matching for e-Commerce. In Proceedings of the Twelfth ACM International Conference on Web Search and Data Mining (Melbourne VIC, Australia) (WSDM'19) . Association for Computing Machinery, New York, NY, USA, 402-410. https://doi.org/10.1145/3289600.3291039 - · DOI
  22. [25][25] Chenyan Xiong, Zhuyun Dai, Jamie Callan, Zhiyuan Liu, and Russell Power. 2017. End-to-End Neural Ad-Hoc Ranking with Kernel Pooling. In Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval (Shinjuku, Tokyo, Japan) (SIGIR '17) . Association for Computing Machinery, New York, NY, USA, 55-64. https://doi.org/10.1145/ 3077136.3080809 -
  23. [26][26] Jun Xu, Xiangnan He, and Hang Li. 2019. Deep Learning for Matching in Search and Recommendation. In Proceedings of the Twelfth ACM International Conference on Web Search and Data Mining (Melbourne VIC, Australia) (WSDM '19) . Association for Computing Machinery, New York, NY, USA, 832-833. https://doi.org/10.1145/3289600.3291380 - · DOI
  24. [27][27] Hongchun Zhang, Tianyi Wang, Xiaonan Meng, and Yi Hu. 2019. Improving Semantic Matching via Multi-Task Learning in E-Commerce. In eCOM@SIGIR . -
  25. [28][28] Guorui Zhou, Xiaoqiang Zhu, Chengru Song, Ying Fan, Han Zhu, Xiao Ma, Yanghui Yan, Junqi Jin, Han Li, and Kun Gai. 2018. Deep Interest Network for Click-Through Rate Prediction. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, KDD 2018, London, UK, August 19-23, 2018 , Yike Guo and Faisal Farooq (Eds.). ACM, 1059-1068. https://doi.org/10.1145/3219819.3219823 · DOI · Paper page

Cited by

Sorted by citations

Authors
Yao, S., Tan, J., Chen, X., Yang, K., Xiao, R., Deng, H., & Wan, X.
Affiliations
  • Peking University (CN)
  • Alibaba Group (China) (CN)
  • Alibaba Group (United States) (US)
ORCID
Venue
Proceedings of the Web Conference 2021
Publisher
ACM
Published
2021-04-19
Pages
2890-2899
DOI
10.1145/3442381.3450129
OpenAlex
W3133376386
Cited by
33
References
3
Citations by year
2026: 3 · 2025: 9 · 2024: 7 · 2023: 8 · 2022: 6
FWCI
4.35
Type
conference paper
Language
en
Open access
Licence
Locations
  • International World Wide Web Conference (WWW) (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
  • Physical Sciences › Computer Science › Information Systems › Recommender Systems and Techniques
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
Keywords
click-through data, e-commerce search, weakly supervised learning, click-through rate prediction, semantic relevance, score distribution, search engine ranking, Taobao

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review