Student or educator? Sign up with your school email and get 6 months of Pro free.
Cited by 1,568

The DOI matches its Crossref record; 7 of the 7 references it cites were found in a registry or an index.

Learning to Rank for Information Retrieval

Liu, Tie-Yan·2009·Foundations and Trends® in Information Retrieval

Abstract

Learning to rank for Information Retrieval (IR) is a task to automatically construct a ranking model using training data, such that the model can sort new objects according to their degrees of relevance, preference, or importance. Many IR problems are by nature ranking problems, and many IR technologies can be potentially enhanced by using learning-to-rank techniques. The objective of this tutorial is to give an introduction to this research direction. Specifically, the existing learning-to-rank algorithms are reviewed and categorized into three approaches: the pointwise, pairwise, and listwise approaches. The advantages and disadvantages with each approach are analyzed, and the relationships between the loss functions used in these approaches and IR evaluation measures are discussed. Then the empirical evaluations on typical learning-to-rank methods are shown, with the LETOR collection as a benchmark dataset, which seems to suggest that the listwise approach be the most effective one among all the approaches. After that, a statistical ranking theory is introduced, which can describe different learning-to-rank algorithms, and be used to analyze their query-level generalization abilities. At the end of the tutorial, we provide a summary and discuss potential future work on learning to rank.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1561/1500000016

Cites

7 of 7 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Research works printed but not linked: 131

  1. Agarwal, (2005), Generalization bounds for the area under the ROC curve, Journal of Machine Learning, 6, 393
  2. Agarwal, (2005), Stability and generalization of bipartite ranking algorithms, 32
  3. Agichtein, (2006), Learning user interaction models for predicting web search result preferences, 3, 10.1145/1148170.1148175 · DOI · Paper page
  4. Almeida, (2007), A combined component approach for finding collection-adapted ranking functions based on genetic programming, 399, 10.1145/1277741.1277810 · DOI
  5. Amini, (2008), A boosting algorithm for learning bipartite ranking functions with partially labeled data, 99
  6. Andrews, (2003), Support vector machines for multiple-instance learning, 561
  7. Aslam, (2001), Models for metasearch, 276, 10.1145/383952.384007 · DOI · Paper page
  8. Baeza-Yates, (1999), Modern Information Retrieval
  9. Bartell, (1995), Learning to retrieve information
  10. Bartlett, (2002), Rademacher and Gaussian complexities risk bounds and structural results, Journal of Machine Learning, 463
  11. Bousquet, (2004), Introduction to statistical learning theory, 169, 10.1007/978-3-540-28650-9_8 · DOI · Paper page
  12. Bousquet, (2002), Stability and generalization, The Journal of Machine Learning Research, 2, 449
  13. Burges, (2006), Learning to rank with nonsmooth cost functions, 395
  14. Cao, (2007), Learning to rank documents for ad-hoc retrieval with regularized models
  15. Cao, (2006), Adapting ranking SVM to document retrieval, 186, 10.1145/1148170.1148205 · DOI · Paper page
  16. Carterette, (2008), Evaluation over thousands of queries, 651
  17. Carvalho, (2008), A metaleaning approach for robust rank learning
  18. Chakrabarti, (2008), Structured learning for non-smooth ranking losses, 88
  19. Chapelle, (2007), Large margin optimization of ranking measures
  20. Chirita, (2005), MailRank: Using ranking for spam detection, 373, 10.1145/1099554.1099671 · DOI
  21. Chu, (2005), Gaussian processes for ordinal regression, Journal of Machine Learning Research, 6, 1019
  22. Chu, (2005), Preference learning with Gaussian processes, 137, 10.1145/1102351.1102369 · DOI · Paper page
  23. Chu, (2005), New approaches to support vector ordinal regression, 145, 10.1145/1102351.1102370 · DOI · Paper page
  24. Clemencon, (2008), Ranking and empirical minimization of U-statistics, The Annals of Statistics, 36, 844, 10.1214/009052607000000910 · DOI · Paper page
  25. Clemencon, (2007), Ranking the best instances, Journal of Machine Learning Research, 8, 2671
  26. Cohen, (1998), Learning to order things, 243
  27. Collins, (2002), Ranking algorithms for named-entity extraction: Boosting and the voted perceptron, 7
  28. Cooper, (1992), Probabilistic retrieval based on staged logistic regression, 198
  29. Cortes, (2007), Magnitude-preserving ranking algorithms, 169, 10.1145/1273496.1273518 · DOI
  30. Cossock, (2006), Subset ranking using regression, 605
  31. Crammer, (2002), Pranking with ranking, 641
  32. Craswell, (2003), Overview of the TREC 2003 Web track, 78
  33. Dave, (2003), Mining the peanut gallery: Opinion extraction and semantic classification of product reviews, 519
  34. Deerwester, (1990), Indexing by latent semantic analysis, Journal of the American Society for Information Science, 41, 391, 10.1002/(SICI)1097-4571(199009)41:6<391::AID-ASI1>3.0.CO;2-9 · DOI · Paper page
  35. Diaz, (2007), Regularizing query-based retrieval scores, Information Retrieval, 10, 531, 10.1007/s10791-007-9034-8 · doi:10.1007/s10791-007-9034-8
  36. Drucker, (2002), Support vector machines: Relevance feedback and information retrieval, Information Processing and Management, 38, 305, 10.1016/S0306-4573(01)00037-1 · doi:10.1016/s0306-4573(01)00037-1
  37. Duh, (2008), Learning to rank with partially-labeled data, 251
  38. Fan, (2004), The effects of fitness functions on genetic programming based ranking discovery for web search, Journal of American Society for Information Science and Technology, 55, 628, 10.1002/asi.20009 · DOI
  39. Fan, (2004), Discovery of context-specific ranking functions for effective information retrieval using genetic programming, IEEE Transactions on Knowledge and Data Engineering, 16, 523, 10.1109/TKDE.2004.1269663 · DOI
  40. Fan, (2004), A generic ranking function discovery framework by genetic programming for information retrieval, Information Processing and Management, 40, 587, 10.1016/j.ipm.2003.08.001 · DOI
  41. Fan, (2005), Genetic programming-based discovery of ranking functions for effective web search, Journal of Management of Information Systems, 21, 37, 10.1080/07421222.2005.11045828 · DOI
  42. Fan, (2006), On linear mixture of expert approaches to information retrieval, Decision Support System, 42, 975, 10.1016/j.dss.2004.11.014 · DOI
  43. Fan, (2004), Ranking function optimization for effective web search by genetic programming: An empirical study, 40105
  44. Freund, (2003), An efficient boosting algorithm for combining preferences, Journal of Machine Learning Research, 4, 933
  45. Fuhr, (1989), Optimum polynomial retrieval functions based on the probability ranking principle, ACM Transactions on Information Systems, 7, 183, 10.1145/65943.65944 · DOI
  46. Fung, (2005), Learning rankings via convex hull separation
  47. Geng, (2007), Feature selection for ranking, 407, 10.1145/1277741.1277811 · DOI · Paper page
  48. Geng, (2008), Query-dependent ranking using k-nearest neighbor, 115
  49. Gey, (1994), Inferring probability of relevance using the method of logistic regression, 222
  50. Guiasu, (1985), The principle of maximum entropy, The Mathematical Intelligencer, 7, 42, 10.1007/BF03023004 · DOI · Paper page
  51. Guiver, (2008), Learning to rank with softrank and Gaussian processes, 259
  52. Gyöngyi, (2004), Combating web spam with trustrank, 576
  53. Gyongyi, (2004), Combating web spam with trustrank, 576
  54. Harrington, (2003), Online ranking/collaborative filtering using the perceptron algorithm, 250
  55. Harrington, (2003), Online ranking/collaborative filtering using the perceptron algorithm, 250
  56. He, (2003), A study of parameter tuning for term frequency normalization, 10
  57. Herbrich, (2001), Bayes point machines, Journal of Machine Learning Research, 1, 245
  58. Herbrich, (2000), Large margin rank boundaries for ordinal regression, 115, 10.7551/mitpress/1113.003.0010 · DOI · Paper page
  59. Hersh, (1994), OHSUMED: An interactive retrieval evaluation and new large test collection for research, 192
  60. Järvelin, (2000), IR evaluation methods for retrieving highly relevant documents, 41, 10.1145/345508.345545 · DOI · Paper page
  61. Jin, (2008), Ranking refinement and its application to information retrieval, 397
  62. Joachims, (2002), Optimizing search engines using clickthrough data, 133
  63. Joachims, (2003), Text Mining, 79
  64. Joachims, (2005), A support vector method for multivariate performance measures, 377
  65. Kang, (2003), Query type classification for web document retrieval, 64, 10.1145/860435.860449 · DOI · Paper page
  66. Kleinberg, (1999), Authoritative sources in a hyperlinked environment, Journal of ACM, 46, 604, 10.1145/324133.324140 · DOI · Paper page
  67. Kramer, (2000), Prediction of ordinal classes using regression trees, Funfamenta Informaticae, 34, 1
  68. Lafferty, (2001), Document language models, query models and risk minimization for information retrieval, 111, 10.1145/383952.383970 · DOI · Paper page
  69. Lan, (2009), Generalization analysis of listwise learning-to-rank algorithms, 10.1145/1553374.1553449 · DOI
  70. Lan, (2008), Query-level stability and generalization in learning to rank, 512, 10.1145/1390156.1390221 · DOI
  71. Li, (2003), A loss function analysis for classification methods in text categorization, 472
  72. Li, (2007), McRank: Learning to rank using multiple classification and gradient boosting, 845
  73. Liu, (2007), LETOR: Benchmark dataset for research on learning to rank for information retrieval, 10.1561/9781601982452 · DOI
  74. Liu, (2007), Supervised rank aggregation, 481
  75. Luce, (1959), Individual Choice Behavior
  76. Mallows, (1975), Non-null ranking models, Biometrika, 44, 114, 10.1093/biomet/44.1-2.114 · DOI · Paper page
  77. Maron, (1960), On relevance, probabilistic indexing and information retrieval, Journal of the ACM, 7, 216, 10.1145/321033.321035 · DOI · Paper page
  78. Matveeva, (2006), High accuracy retrieval with multiple nested ranker, 437, 10.1145/1148170.1148246 · DOI
  79. Metzler, (2005), A Markov random field model for term dependencies, 472, 10.1145/1076034.1076115 · DOI · Paper page
  80. Metzler, (2008), Machine learned sentence selection strategies for query-biased summarization
  81. Minka, (2008), Selection bias in the LETOR datasets
  82. Mitchell, (1997), Machine Learning
  83. Nallapati, (2004), Discriminative models for information retrieval, 64, 10.1145/1008992.1009006 · DOI · Paper page
  84. Nie, (2006), Topical link analysis for web search, 91, 10.1145/1148170.1148189 · DOI
  85. Pahikkala, (2007), Learning to rank with pairwise regularized least-squares
  86. Pang, (2005), Seeing stars: Exploiting class relationships for sentiment categorization with respect to rating scales, 115
  87. Qin, (2007), Ranking with multiple hyperplanes, 279, 10.1145/1277741.1277791 · DOI
  88. Qin, (2008), How to make LETOR more useful and reliable
  89. Qin, (2005), A study of relevance propagation for web search, 408, 10.1145/1076034.1076105 · DOI
  90. Qin, (2007), Query-level loss functions for information retrieval, Information Processing and Management, 44, 838, 10.1016/j.ipm.2007.07.016 · DOI
  91. Qin, (2008), Learning to rank relational objects and its application to web search, 407
  92. Qin, (2008), Global ranking using continuous conditional random fields, 1281
  93. Radlinski, (2005), Query chain: Learning to rank from implicit feedback, 239, 10.1145/1081870.1081899 · DOI · Paper page
  94. Radlinski, (2007), Active exploration for learning rankings from clickthrough data, 10.1145/1281192.1281254 · DOI
  95. Radlinski, (2008), Learning diverse rankings with multi-armed bandits, 784, 10.1145/1390156.1390255 · DOI · Paper page
  96. Rajaram, (2005), Generalization bounds for k-partite ranking
  97. Rigutini, (2008), Learning to rank by a neural-based sorting algorithm
  98. Robertson, (2007), On rank-based effectiveness measures and optimization, Information Retrieval, 10, 321, 10.1007/s10791-007-9025-9 · DOI
  99. Robertson, (1997), Overview of the okapi projects, Journal of Documentation, 53, 3, 10.1108/EUM0000000007186 · DOI · Paper page
  100. Rocchio, (1971), The SMART Retrieval System — Experiments in Automatic Document Processing, 313
  101. Shakery, (2006), A probabilistic relevance propagation model for hypertext retrieval, 550
  102. Shashua, (2002), Ranking with large margin principles: Two approaches, 937
  103. Shawe-Taylor, (2004), Kernel Methods for Pattern Analysis, 10.1017/CBO9780511809682 · DOI · Paper page
  104. Singhal, (2001), Modern information retrieval: A brief overview, IEEE Data Engineering Bulletin, 24, 35
  105. Tao, (2006), Regularized estimation of mixture models for robust pseudo-relevance feedback, 162, 10.1145/1148170.1148201 · doi:10.1145/1148170.1148201
  106. Tao, (2007), An exploration of proximity measures in information retrieval, 295, 10.1145/1277741.1277794 · DOI · Paper page
  107. Taylor, (2008), SoftRank: Optimising non-smooth rank metrics, 77, 10.1145/1341531.1341544 · DOI · Paper page
  108. Taylor, (2006), Optimisation methods for ranking functions with multiple parameters, 585
  109. Trotman, (2005), Learning to rank, Information Retrieval, 8, 359, 10.1007/s10791-005-6991-7 · DOI
  110. Tsai, (2007), FRank: A ranking method with fidelity loss, 383, 10.1145/1277741.1277808 · DOI · Paper page
  111. Tsochantaridis, (2004), Support vector machine learning for interdependent and structured output space, 104, 10.1145/1015330.1015341 · DOI · Paper page
  112. Usunier, (2005), Generalization error bounds for classifiers trained with interdependent data, 1369
  113. Vapnik, (1998), Statistical Learning Theory
  114. Veloso, (2008), Learning to rank at query-time using association rules, 267
  115. Xi, (2004), Learning effective ranking functions for newsgroup search, 394, 10.1145/1008992.1009060 · DOI
  116. Xia, (2008), Listwise approach to learning to rank — Theorem and algorithm, 1192, 10.1145/1390156.1390306 · DOI · Paper page
  117. Xu, (2005), Ranking definitions with supervised learning methods, 811
  118. Xu, (2007), AdaRank: A boosting algorithm for information retrieval, 391, 10.1145/1277741.1277809 · DOI · Paper page
  119. Xu, (2008), Directly optimizing IR evaluation measures in learning to rank, 107
  120. Xue, (2005), Exploiting the hierarchical structure for link analysis, 186, 10.1145/1076034.1076068 · DOI
  121. Yeh, (2007), Learning to rank for information retrieval using genetic programming
  122. Yu, (2005), SVM selective sampling for ranking with application to data retrieval, 354, 10.1145/1081870.1081911 · DOI
  123. Yue, (2007), A support vector method for optimizing average precision, 271, 10.1145/1277741.1277790 · DOI · Paper page
  124. Yue, (2008), Predicting diverse subsets using structural SVM, 1224, 10.1145/1390156.1390310 · DOI · Paper page
  125. Zhai, (2008), Language models, Foundations and Trends in Information Retrieval
  126. Zhai, (2003), Beyond independent relevance: Methods and evaluation metrics for subtopic retrieval, 10, 10.1145/860435.860440 · DOI · Paper page
  127. Zhai, (2001), Model-based feedback in the language modeling approach to information retrieval, 403
  128. Zhai, (2006), A risk minimization framework for information retrieval, Information Processing and Management, 42, 31, 10.1016/j.ipm.2004.11.003 · DOI
  129. Zheng, (2008), Query-level learning to rank using isotonic regression, 10.1109/ALLERTON.2008.4797684 · DOI
  130. Zhou, (2008), Learning to rank with ties, 275
  131. Zoeter, (2008), A decision theoretic framework for ranking using implicit feedback

Cited by

See all 1,568

Metadata

Authors
Liu, T.-Y.
Venue
Foundations and Trends® in Information Retrieval
Publisher
Emerald
Published
2009-06-27
Volume
3
Issue
3
Pages
225-331
ISSN
  • 1554-0669 (ISSN-L)
  • 1554-0677
OpenAlex
W2149427297
Cited by
1,568
References
7
FWCI
133.29
Type
journal article
Language
en
Open access
closed

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.