The DOI matches its Crossref record; 36 of the 36 references it cites were found in a registry or an index.
Learning to Rank with Selection Bias in Personal Search
Abstract
Click-through data has proven to be a critical resource for improving search ranking quality. Though a large amount of click data can be easily collected by search engines, various biases make it difficult to fully leverage this type of data. In the past, many click models have been proposed and successfully used to estimate the relevance for individual query-document pairs in the context of web search. These click models typically require a large quantity of clicks for each individual pair and this makes them difficult to apply in systems where click data is highly sparse due to personalized corpora and information needs, e.g., personal search. In this paper, we study the problem of how to leverage sparse click data in personal search and introduce a novel selection bias problem and address it in the learning-to-rank framework. This paper proposes a few bias estimation methods, including a novel query-dependent one that captures queries with similar results and can successfully deal with sparse data. We empirically demonstrate that learning-to-rank that accounts for query-dependent selection bias yields significant improvements in search effectiveness through online experiments with one of the world's largest personal search engines.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The central role of the propensity score in observational studies for causal effects
ROSENBAUM, P. R., & RUBIN, D. B. (1983) · Biometrika · doi:10.1093/biomet/70.1.41 · Cited by 31,241
The Elements of Statistical Learning
Hastie, T., Friedman, J., & Tibshirani, R. (2001) · Springer Series in Statistics · doi:10.1007/978-0-387-21606-5 · Cited by 24,162
Optimizing search engines using clickthrough data
Joachims, T. (2002) · Proceedings of the eighth ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/775047.775067 · Cited by 3,873
Learning to rank for information retrieval
Liu, T.-Y. (2010) · Proceedings of the 33rd international ACM SIGIR conference on Research and development in information retrieval · doi:10.1145/1835449.1835676 · Cited by 1,856
Correcting Sample Selection Bias by Unlabeled Data
Huang, J., Smola, A. J., Gretton, A., Borgwardt, K. M., & Schölkopf, B. (2007) · Advances in Neural Information Processing Systems 19 · doi:10.7551/mitpress/7503.003.0080 · Cited by 1,571
Learning to Rank for Information Retrieval
Liu, T.-Y. (2009) · Foundations and Trends® in Information Retrieval · doi:10.1561/1500000016 · Cited by 1,568
Accurately interpreting clickthrough data as implicit feedback
Joachims, T., Granka, L., Pan, B., Hembrooke, H., & Gay, G. (2005) · Proceedings of the 28th annual international ACM SIGIR conference on Research and development in information retrieval · doi:10.1145/1076034.1076063 · Cited by 1,408
Predicting clicks: estimating the click-through rate for new ads
Richardson, M., Dominowska, E., & Ragno, R. (2007) · Proceedings of the 16th international conference on World Wide Web · doi:10.1145/1242572.1242643 · Cited by 1,049
From RankNet to LambdaRank to LambdaMART: An Overview
Burges, C. J. C. (2010) · Cited by 1,011
An experimental comparison of click position-bias models
Craswell, N., Zoeter, O., Taylor, M., & Ramsey, B. (2008) · Proceedings of the international conference on Web search and web data mining - WSDM '08 · doi:10.1145/1341531.1341545 · Cited by 942
Learning and evaluating classifiers under sample selection bias
Zadrozny, B. (2004) · Twenty-first international conference on Machine learning - ICML '04 · doi:10.1145/1015330.1015425 · Cited by 800 · cited 1×
“The methods proposed in this paper can also be viewed as a novel extension of sample selection bias correction methods, which are well-studied in the context of regression and classification [20, 34, 32], in the online learning-to-rank setting.”
MethodStuff I've Seen: A System for Personal Information Retrieval and Re-Use
Dumais, S., Cutrell, E., Cadiz, J. J., Jancke, G., Sarin, R., & Robbins, D. C. (2016) · ACM SIGIR Forum · doi:10.1145/2888422.2888425 · Cited by 691
A dynamic bayesian network click model for web search ranking
Chapelle, O., & Zhang, Y. (2009) · Proceedings of the 18th international conference on World wide web · doi:10.1145/1526709.1526711 · Cited by 537
LETOR: A benchmark collection for research on learning to rank for information retrieval
Qin, T., Liu, T.-Y., Xu, J., & Li, H. (2010) · Information Retrieval · doi:10.1007/s10791-009-9123-y · Cited by 461
Unbiased offline evaluation of contextual-bandit-based news article recommendation algorithms
Li, L., Chu, W., Langford, J., & Wang, X. (2011) · Proceedings of the fourth ACM international conference on Web search and data mining · doi:10.1145/1935826.1935878 · Cited by 443
Self-citationA user browsing model to predict search engine click data from past observations.
Dupret, G. E., & Piwowarski, B. (2008) · Proceedings of the 31st annual international ACM SIGIR conference on Research and development in information retrieval · doi:10.1145/1390334.1390392 · Cited by 442
Chuklin, A., Markov, I., & de Rijke, M. (2015) · Synthesis Lectures on Information Concepts, Retrieval, and Services · doi:10.1007/978-3-031-02294-4 · Cited by 398
Stuff I've seen: a system for personal information retrieval and re-use
Dumais, S., Cutrell, E., Cadiz, J., Jancke, G., Sarin, R., & Robbins, D. C. (2003) · Proceedings of the 26th annual international ACM SIGIR conference on Research and development in informaion retrieval · doi:10.1145/860435.860451 · Cited by 381
Doubly Robust Policy Evaluation and Learning
Dudı́k, M., Langford, J. C., & Li, L. (2011) · arXiv (Cornell University) · doi:10.48550/arxiv.1103.4601 · Cited by 375
An estimate of an upper bound for the entropy of English
Brown, P. F., Pietra, V. J. D., Mercer, R. L., Pietra, S. A. D., & Lai, J. C. (1992) · OLAC - Open Language Archives Community · doi:10.5555/146680.146685 · Cited by 341
Click chain model in web search
Guo, F., Liu, C., Kannan, A., Minka, T., Taylor, M., Wang, Y.-M., & Faloutsos, C. (2009) · Proceedings of the 18th international conference on World wide web · doi:10.1145/1526709.1526712 · Cited by 219
A regression framework for learning ranking functions using relative relevance judgments
Zheng, Z., Chen, K., Sun, G., & Zha, H. (2007) · Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval · doi:10.1145/1277741.1277792 · Cited by 204
Yue, Y., Patel, R., & Roehrig, H. (2010) · Proceedings of the 19th international conference on World wide web · doi:10.1145/1772690.1772793 · Cited by 186
Kamvar, M., Kellar, M., Patel, R., & Xu, Y. (2009) · Proceedings of the 18th international conference on World wide web · doi:10.1145/1526709.1526817 · Cited by 182
Automatic Categorization of Email into Folders: Benchmark Experiments on Enron and SRI Corpora
Bekkerman, R., McCallum, A., & Huang, G. B. (2005) · ScholarWorks@UMassAmherst (University of Massachusetts Amherst) · Cited by 179
A model to estimate intrinsic document relevance from the clickthrough logs of a web search engine
Dupret, G., & Liao, C. Y. (2010) · ACM International Conference on Web Search and Data Mining (WSDM) · doi:10.1145/1718487.1718510 · Cited by 146
Counterfactual Risk Minimization: Learning from Logged Bandit FeedbackPreprint
Swaminathan, A., & Joachims, T. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1502.02362 · Cited by 121
Reliable information retrieval evaluation with incomplete and biased judgements
Büttcher, S., Clarke, C. L. A., Yeung, P. C. K., & Soboroff, I. M. (2007) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/1277741.1277755 · Cited by 120
Evaluating online ad campaigns in a pipeline: causal models at scale
Chan, D. X., Ge, R., Gershony, O., Hesterberg, T., & Lambert, D. (2010) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/1835804.1835809 · Cited by 116
A novel click model and its applications to online advertising
Allen-Zhu, Z., Chen, W., Minka, T., Zhu, C., & Chen, Z. (2010) · ACM International Conference on Web Search and Data Mining (WSDM) · doi:10.1145/1718487.1718528 · Cited by 102
Counterfactual Estimation and Optimization of Click Metrics in Search Engines
Li, L., Chen, S., Kleban, J., & Gupta, A. (2015) · International World Wide Web Conference (WWW) · doi:10.1145/2740908.2742562 · Cited by 82
Balancing Exploration and Exploitation in Learning to Rank Online
Hofmann, K., Whiteson, S., & de Rijke, M. (2011) · Lecture notes in computer science · doi:10.1007/978-3-642-20161-5_25 · Cited by 69
Document selection methodologies for efficient and effective learning-to-rank
Aslam, J. A., Kanoulas, E., Pavlu, V., Savev, S., & Yılmaz, E. (2009) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/1571941.1572022 · Cited by 66
Keane, M. T., & O’Brien, M. (2006) · eScholarship (California Digital Library) · Cited by 32
Selection Bias in the LETOR Datasets
Minka, T. (2008) · Cited by 25
A Whole Page Click Model to Better Interpret Search Engine Click Data
Chen, W., Ji, Z., Shen, S., & Yang, Q. (2011) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v25i1.8084 · Cited by 20
Cited by
Bias and Debias in Recommender System: A Survey and Future Directions
Chen, J., Dong, H., Wang, X., Feng, F., Wang, M., & He, X. (2023) · ACM Transactions on Information Systems · doi:10.1145/3564284 · Cited by 741
“…leveraged counterfactual reasoning to eliminate the direct causal effect from exposure features on the prediction; Yang et al. [191] mitigates exposure bias through counterfactual samples; Liu et al. [108] disentangle the effect from exposure and preference with introducing information bottleneck.”
BackgroundInfluentialSelf-citationUnbiased Learning-to-Rank with Biased Feedback
Joachims, T., Swaminathan, A., & Schnabel, T. (2017) · Proceedings of the Tenth ACM International Conference on Web Search and Data Mining · doi:10.1145/3018661.3018699 · Cited by 531
“Finally, our approach is similar in spirit to [28], where propensity-weighting is used to correct for selection bias when discarding queries without clicks during learning-torank.”
BackgroundMethodInfluentialMicrosoft Academic Graph: When experts are not enough
Wang, K., Shen, Z., Huang, C., Wu, C.-H., Dong, Y., & Kanakia, A. (2020) · Quantitative Science Studies · doi:10.1162/qss_a_00021 · Cited by 509
Equity of Attention: Amortizing Individual Fairness in Rankings
Biega, A. J., Gummadi, K. P., & Weikum, G. (2018) · The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval · doi:10.1145/3209978.3210063 · Cited by 337
Recommending what video to watch next: a multitask ranking system
Zhao, Z., Hong, L., Wei, L., Chen, J., Nath, A., Andrews, S., Kumthekar, A., Sathiamoorthy, M., Yi, X., & Chi, E. (2019) · Proceedings of the 13th ACM Conference on Recommender Systems · doi:10.1145/3298689.3346997 · Cited by 311
Metadata
- Authors
- Wang, X., Bendersky, M., Metzler, D., & Najork, M.
- Venue
- Proceedings of the 39th International ACM SIGIR conference on Research and Development in Information Retrieval
- Publisher
- ACM
- Published
- 2016-07-07
- Pages
- 115-124
- OpenAlex
- W2340526403
- Cited by
- 284
- References
- 36
- FWCI
- 18.80
- Type
- conference paper
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.