The DOI matches its Crossref record; 7 of the 7 references it cites were found in a registry or an index.
Learning to Rank for Information Retrieval
Abstract
Learning to rank for Information Retrieval (IR) is a task to automatically construct a ranking model using training data, such that the model can sort new objects according to their degrees of relevance, preference, or importance. Many IR problems are by nature ranking problems, and many IR technologies can be potentially enhanced by using learning-to-rank techniques. The objective of this tutorial is to give an introduction to this research direction. Specifically, the existing learning-to-rank algorithms are reviewed and categorized into three approaches: the pointwise, pairwise, and listwise approaches. The advantages and disadvantages with each approach are analyzed, and the relationships between the loss functions used in these approaches and IR evaluation measures are discussed. Then the empirical evaluations on typical learning-to-rank methods are shown, with the LETOR collection as a benchmark dataset, which seems to suggest that the listwise approach be the most effective one among all the approaches. After that, a statistical ranking theory is introduced, which can describe different learning-to-rank algorithms, and be used to analyze their query-level generalization abilities. At the end of the tutorial, we provide a summary and discuss potential future work on learning to rank.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The Nature of Statistical Learning Theory
Vapnik, V. N. (1995) · doi:10.1007/978-1-4757-2440-0 · Cited by 39,138
A decision-theoretic generalization of on-line learning and an application to boosting
Freund, Y., & Schapire, R. E. (1997) · Journal of Computer and System Sciences · doi:10.1006/jcss.1997.1504 · Cited by 20,466
Cumulated gain-based evaluation of IR techniques
Järvelin, K., & Kekäläinen, J. (2002) · ACM Transactions on Information Systems · doi:10.1145/582415.582418 · Cited by 4,841
Learning to rank using gradient descent
Burges, C., Shaked, T., Renshaw, E., Lazier, A., Deeds, M., Hamilton, N., & Hullender, G. (2005) · Proceedings of the 22nd international conference on Machine learning - ICML '05 · doi:10.1145/1102351.1102363 · Cited by 2,836
Learning to rank: from pairwise approach to listwise approach
Cao, Z., Qin, T., Liu, T.-Y., Tsai, M.-F., & Li, H. (2007) · Proceedings of the 24th international conference on Machine learning · doi:10.1145/1273496.1273513 · Cited by 1,960
Self-citationA language modeling approach to information retrieval
Ponte, J. M., & Croft, W. B. (1998) · Proceedings of the 21st annual international ACM SIGIR conference on Research and development in information retrieval · doi:10.1145/290941.291008 · Cited by 1,821
Plackett, R. L. (1975) · Applied Statistics · doi:10.2307/2346567 · Cited by 731
Research works printed but not linked: 131
- Agarwal, (2005), Generalization bounds for the area under the ROC curve, Journal of Machine Learning, 6, 393
- Agarwal, (2005), Stability and generalization of bipartite ranking algorithms, 32
- Agichtein, (2006), Learning user interaction models for predicting web search result preferences, 3, 10.1145/1148170.1148175 · DOI · Paper page
- Almeida, (2007), A combined component approach for finding collection-adapted ranking functions based on genetic programming, 399, 10.1145/1277741.1277810 · DOI
- Amini, (2008), A boosting algorithm for learning bipartite ranking functions with partially labeled data, 99
- Andrews, (2003), Support vector machines for multiple-instance learning, 561
- Aslam, (2001), Models for metasearch, 276, 10.1145/383952.384007 · DOI · Paper page
- Baeza-Yates, (1999), Modern Information Retrieval
- Bartell, (1995), Learning to retrieve information
- Bartlett, (2002), Rademacher and Gaussian complexities risk bounds and structural results, Journal of Machine Learning, 463
- Bousquet, (2004), Introduction to statistical learning theory, 169, 10.1007/978-3-540-28650-9_8 · DOI · Paper page
- Bousquet, (2002), Stability and generalization, The Journal of Machine Learning Research, 2, 449
- Burges, (2006), Learning to rank with nonsmooth cost functions, 395
- Cao, (2007), Learning to rank documents for ad-hoc retrieval with regularized models
- Cao, (2006), Adapting ranking SVM to document retrieval, 186, 10.1145/1148170.1148205 · DOI · Paper page
- Carterette, (2008), Evaluation over thousands of queries, 651
- Carvalho, (2008), A metaleaning approach for robust rank learning
- Chakrabarti, (2008), Structured learning for non-smooth ranking losses, 88
- Chapelle, (2007), Large margin optimization of ranking measures
- Chirita, (2005), MailRank: Using ranking for spam detection, 373, 10.1145/1099554.1099671 · DOI
- Chu, (2005), Gaussian processes for ordinal regression, Journal of Machine Learning Research, 6, 1019
- Chu, (2005), Preference learning with Gaussian processes, 137, 10.1145/1102351.1102369 · DOI · Paper page
- Chu, (2005), New approaches to support vector ordinal regression, 145, 10.1145/1102351.1102370 · DOI · Paper page
- Clemencon, (2008), Ranking and empirical minimization of U-statistics, The Annals of Statistics, 36, 844, 10.1214/009052607000000910 · DOI · Paper page
- Clemencon, (2007), Ranking the best instances, Journal of Machine Learning Research, 8, 2671
- Cohen, (1998), Learning to order things, 243
- Collins, (2002), Ranking algorithms for named-entity extraction: Boosting and the voted perceptron, 7
- Cooper, (1992), Probabilistic retrieval based on staged logistic regression, 198
- Cortes, (2007), Magnitude-preserving ranking algorithms, 169, 10.1145/1273496.1273518 · DOI
- Cossock, (2006), Subset ranking using regression, 605
- Crammer, (2002), Pranking with ranking, 641
- Craswell, (2003), Overview of the TREC 2003 Web track, 78
- Dave, (2003), Mining the peanut gallery: Opinion extraction and semantic classification of product reviews, 519
- Deerwester, (1990), Indexing by latent semantic analysis, Journal of the American Society for Information Science, 41, 391, 10.1002/(SICI)1097-4571(199009)41:6<391::AID-ASI1>3.0.CO;2-9 · DOI · Paper page
- Diaz, (2007), Regularizing query-based retrieval scores, Information Retrieval, 10, 531, 10.1007/s10791-007-9034-8 · doi:10.1007/s10791-007-9034-8
- Drucker, (2002), Support vector machines: Relevance feedback and information retrieval, Information Processing and Management, 38, 305, 10.1016/S0306-4573(01)00037-1 · doi:10.1016/s0306-4573(01)00037-1
- Duh, (2008), Learning to rank with partially-labeled data, 251
- Fan, (2004), The effects of fitness functions on genetic programming based ranking discovery for web search, Journal of American Society for Information Science and Technology, 55, 628, 10.1002/asi.20009 · DOI
- Fan, (2004), Discovery of context-specific ranking functions for effective information retrieval using genetic programming, IEEE Transactions on Knowledge and Data Engineering, 16, 523, 10.1109/TKDE.2004.1269663 · DOI
- Fan, (2004), A generic ranking function discovery framework by genetic programming for information retrieval, Information Processing and Management, 40, 587, 10.1016/j.ipm.2003.08.001 · DOI
- Fan, (2005), Genetic programming-based discovery of ranking functions for effective web search, Journal of Management of Information Systems, 21, 37, 10.1080/07421222.2005.11045828 · DOI
- Fan, (2006), On linear mixture of expert approaches to information retrieval, Decision Support System, 42, 975, 10.1016/j.dss.2004.11.014 · DOI
- Fan, (2004), Ranking function optimization for effective web search by genetic programming: An empirical study, 40105
- Freund, (2003), An efficient boosting algorithm for combining preferences, Journal of Machine Learning Research, 4, 933
- Fuhr, (1989), Optimum polynomial retrieval functions based on the probability ranking principle, ACM Transactions on Information Systems, 7, 183, 10.1145/65943.65944 · DOI
- Fung, (2005), Learning rankings via convex hull separation
- Geng, (2007), Feature selection for ranking, 407, 10.1145/1277741.1277811 · DOI · Paper page
- Geng, (2008), Query-dependent ranking using k-nearest neighbor, 115
- Gey, (1994), Inferring probability of relevance using the method of logistic regression, 222
- Guiasu, (1985), The principle of maximum entropy, The Mathematical Intelligencer, 7, 42, 10.1007/BF03023004 · DOI · Paper page
- Guiver, (2008), Learning to rank with softrank and Gaussian processes, 259
- Gyöngyi, (2004), Combating web spam with trustrank, 576
- Gyongyi, (2004), Combating web spam with trustrank, 576
- Harrington, (2003), Online ranking/collaborative filtering using the perceptron algorithm, 250
- Harrington, (2003), Online ranking/collaborative filtering using the perceptron algorithm, 250
- He, (2003), A study of parameter tuning for term frequency normalization, 10
- Herbrich, (2001), Bayes point machines, Journal of Machine Learning Research, 1, 245
- Herbrich, (2000), Large margin rank boundaries for ordinal regression, 115, 10.7551/mitpress/1113.003.0010 · DOI · Paper page
- Hersh, (1994), OHSUMED: An interactive retrieval evaluation and new large test collection for research, 192
- Järvelin, (2000), IR evaluation methods for retrieving highly relevant documents, 41, 10.1145/345508.345545 · DOI · Paper page
- Jin, (2008), Ranking refinement and its application to information retrieval, 397
- Joachims, (2002), Optimizing search engines using clickthrough data, 133
- Joachims, (2003), Text Mining, 79
- Joachims, (2005), A support vector method for multivariate performance measures, 377
- Kang, (2003), Query type classification for web document retrieval, 64, 10.1145/860435.860449 · DOI · Paper page
- Kleinberg, (1999), Authoritative sources in a hyperlinked environment, Journal of ACM, 46, 604, 10.1145/324133.324140 · DOI · Paper page
- Kramer, (2000), Prediction of ordinal classes using regression trees, Funfamenta Informaticae, 34, 1
- Lafferty, (2001), Document language models, query models and risk minimization for information retrieval, 111, 10.1145/383952.383970 · DOI · Paper page
- Lan, (2009), Generalization analysis of listwise learning-to-rank algorithms, 10.1145/1553374.1553449 · DOI
- Lan, (2008), Query-level stability and generalization in learning to rank, 512, 10.1145/1390156.1390221 · DOI
- Li, (2003), A loss function analysis for classification methods in text categorization, 472
- Li, (2007), McRank: Learning to rank using multiple classification and gradient boosting, 845
- Liu, (2007), LETOR: Benchmark dataset for research on learning to rank for information retrieval, 10.1561/9781601982452 · DOI
- Liu, (2007), Supervised rank aggregation, 481
- Luce, (1959), Individual Choice Behavior
- Mallows, (1975), Non-null ranking models, Biometrika, 44, 114, 10.1093/biomet/44.1-2.114 · DOI · Paper page
- Maron, (1960), On relevance, probabilistic indexing and information retrieval, Journal of the ACM, 7, 216, 10.1145/321033.321035 · DOI · Paper page
- Matveeva, (2006), High accuracy retrieval with multiple nested ranker, 437, 10.1145/1148170.1148246 · DOI
- Metzler, (2005), A Markov random field model for term dependencies, 472, 10.1145/1076034.1076115 · DOI · Paper page
- Metzler, (2008), Machine learned sentence selection strategies for query-biased summarization
- Minka, (2008), Selection bias in the LETOR datasets
- Mitchell, (1997), Machine Learning
- Nallapati, (2004), Discriminative models for information retrieval, 64, 10.1145/1008992.1009006 · DOI · Paper page
- Nie, (2006), Topical link analysis for web search, 91, 10.1145/1148170.1148189 · DOI
- Pahikkala, (2007), Learning to rank with pairwise regularized least-squares
- Pang, (2005), Seeing stars: Exploiting class relationships for sentiment categorization with respect to rating scales, 115
- Qin, (2007), Ranking with multiple hyperplanes, 279, 10.1145/1277741.1277791 · DOI
- Qin, (2008), How to make LETOR more useful and reliable
- Qin, (2005), A study of relevance propagation for web search, 408, 10.1145/1076034.1076105 · DOI
- Qin, (2007), Query-level loss functions for information retrieval, Information Processing and Management, 44, 838, 10.1016/j.ipm.2007.07.016 · DOI
- Qin, (2008), Learning to rank relational objects and its application to web search, 407
- Qin, (2008), Global ranking using continuous conditional random fields, 1281
- Radlinski, (2005), Query chain: Learning to rank from implicit feedback, 239, 10.1145/1081870.1081899 · DOI · Paper page
- Radlinski, (2007), Active exploration for learning rankings from clickthrough data, 10.1145/1281192.1281254 · DOI
- Radlinski, (2008), Learning diverse rankings with multi-armed bandits, 784, 10.1145/1390156.1390255 · DOI · Paper page
- Rajaram, (2005), Generalization bounds for k-partite ranking
- Rigutini, (2008), Learning to rank by a neural-based sorting algorithm
- Robertson, (2007), On rank-based effectiveness measures and optimization, Information Retrieval, 10, 321, 10.1007/s10791-007-9025-9 · DOI
- Robertson, (1997), Overview of the okapi projects, Journal of Documentation, 53, 3, 10.1108/EUM0000000007186 · DOI · Paper page
- Rocchio, (1971), The SMART Retrieval System — Experiments in Automatic Document Processing, 313
- Shakery, (2006), A probabilistic relevance propagation model for hypertext retrieval, 550
- Shashua, (2002), Ranking with large margin principles: Two approaches, 937
- Shawe-Taylor, (2004), Kernel Methods for Pattern Analysis, 10.1017/CBO9780511809682 · DOI · Paper page
- Singhal, (2001), Modern information retrieval: A brief overview, IEEE Data Engineering Bulletin, 24, 35
- Tao, (2006), Regularized estimation of mixture models for robust pseudo-relevance feedback, 162, 10.1145/1148170.1148201 · doi:10.1145/1148170.1148201
- Tao, (2007), An exploration of proximity measures in information retrieval, 295, 10.1145/1277741.1277794 · DOI · Paper page
- Taylor, (2008), SoftRank: Optimising non-smooth rank metrics, 77, 10.1145/1341531.1341544 · DOI · Paper page
- Taylor, (2006), Optimisation methods for ranking functions with multiple parameters, 585
- Trotman, (2005), Learning to rank, Information Retrieval, 8, 359, 10.1007/s10791-005-6991-7 · DOI
- Tsai, (2007), FRank: A ranking method with fidelity loss, 383, 10.1145/1277741.1277808 · DOI · Paper page
- Tsochantaridis, (2004), Support vector machine learning for interdependent and structured output space, 104, 10.1145/1015330.1015341 · DOI · Paper page
- Usunier, (2005), Generalization error bounds for classifiers trained with interdependent data, 1369
- Vapnik, (1998), Statistical Learning Theory
- Veloso, (2008), Learning to rank at query-time using association rules, 267
- Xi, (2004), Learning effective ranking functions for newsgroup search, 394, 10.1145/1008992.1009060 · DOI
- Xia, (2008), Listwise approach to learning to rank — Theorem and algorithm, 1192, 10.1145/1390156.1390306 · DOI · Paper page
- Xu, (2005), Ranking definitions with supervised learning methods, 811
- Xu, (2007), AdaRank: A boosting algorithm for information retrieval, 391, 10.1145/1277741.1277809 · DOI · Paper page
- Xu, (2008), Directly optimizing IR evaluation measures in learning to rank, 107
- Xue, (2005), Exploiting the hierarchical structure for link analysis, 186, 10.1145/1076034.1076068 · DOI
- Yeh, (2007), Learning to rank for information retrieval using genetic programming
- Yu, (2005), SVM selective sampling for ranking with application to data retrieval, 354, 10.1145/1081870.1081911 · DOI
- Yue, (2007), A support vector method for optimizing average precision, 271, 10.1145/1277741.1277790 · DOI · Paper page
- Yue, (2008), Predicting diverse subsets using structural SVM, 1224, 10.1145/1390156.1390310 · DOI · Paper page
- Zhai, (2008), Language models, Foundations and Trends in Information Retrieval
- Zhai, (2003), Beyond independent relevance: Methods and evaluation metrics for subtopic retrieval, 10, 10.1145/860435.860440 · DOI · Paper page
- Zhai, (2001), Model-based feedback in the language modeling approach to information retrieval, 403
- Zhai, (2006), A risk minimization framework for information retrieval, Information Processing and Management, 42, 31, 10.1016/j.ipm.2004.11.003 · DOI
- Zheng, (2008), Query-level learning to rank using isotonic regression, 10.1109/ALLERTON.2008.4797684 · DOI
- Zhou, (2008), Learning to rank with ties, 275
- Zoeter, (2008), A decision theoretic framework for ranking using implicit feedback
Cited by
Survey on deep learning with class imbalance
Johnson, J. M., & Khoshgoftaar, T. M. (2019) · Journal of Big Data · doi:10.1186/s40537-019-0192-5 · Cited by 3,032
Parikh, D., & Grauman, K. (2011) · 2011 International Conference on Computer Vision · doi:10.1109/iccv.2011.6126281 · Cited by 851
Spotting fake reviewer groups in consumer reviews
Mukherjee, A., Liu, B., & Glance, N. (2012) · Proceedings of the 21st international conference on World Wide Web · doi:10.1145/2187836.2187863 · Cited by 773
Learning to Navigate for Fine-Grained Classification
Yang, Z., Luo, T., Wang, D., Hu, Z., Gao, J., & Wang, L. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01264-9_26 · Cited by 616
Unbiased Learning-to-Rank with Biased Feedback
Joachims, T., Swaminathan, A., & Schnabel, T. (2017) · Proceedings of the Tenth ACM International Conference on Web Search and Data Mining · doi:10.1145/3018661.3018699 · Cited by 531
Metadata
- Authors
- Liu, T.-Y.
- Venue
- Foundations and Trends® in Information Retrieval
- Publisher
- Emerald
- Published
- 2009-06-27
- Volume
- 3
- Issue
- 3
- Pages
- 225-331
- ISSN
- 1554-0669 (ISSN-L)
- 1554-0677
- OpenAlex
- W2149427297
- Cited by
- 1,568
- References
- 7
- FWCI
- 133.29
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.