The DOI matches its Crossref record; 140 of the 140 references it cites were found in a registry or an index.
Learning to Rank for Information Retrieval
Abstract
Learning to rank for Information Retrieval (IR) is a task to automatically construct a ranking model using training data, such that the model can sort new objects according to their degrees of relevance, preference, or importance. Many IR problems are by nature ranking problems, and many IR technologies can be potentially enhanced by using learning-to-rank techniques. The objective of this tutorial is to give an introduction to this research direction. Specifically, the existing learning-to-rank algorithms are reviewed and categorized into three approaches: the pointwise, pairwise, and listwise approaches. The advantages and disadvantages with each approach are analyzed, and the relationships between the loss functions used in these approaches and IR evaluation measures are discussed. Then the empirical evaluations on typical learning-to-rank methods are shown, with the LETOR collection as a benchmark dataset, which seems to suggest that the listwise approach be the most effective one among all the approaches. After that, a statistical ranking theory is introduced, which can describe different learning-to-rank algorithms, and be used to analyze their query-level generalization abilities. At the end of the tutorial, we provide a summary and discuss potential future work on learning to rank.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The Nature of Statistical Learning Theory
Vapnik, V. N. (1995) · doi:10.1007/978-1-4757-2440-0 · Cited by 39,138
Wu, Y. (1999) · Technometrics · doi:10.1080/00401706.1999.10485951 · Cited by 27,221
A decision-theoretic generalization of on-line learning and an application to boosting
Freund, Y., & Schapire, R. E. (1997) · Journal of Computer and System Sciences · doi:10.1006/jcss.1997.1504 · Cited by 20,466
(1999) · Cited by 12,854
Indexing by latent semantic analysis
Deerwester, S., Dumais, S. T., Furnas, G. W., Landauer, T. K., & Harshman, R. (1990) · Journal of the American Society for Information Science · doi:10.1002/(sici)1097-4571(199009)41:6<391::aid-asi1>3.0.co;2-9 · Cited by 12,751
Baeza-Yates, R. A., & Ribeiro-Neto, B. A. (1999) · Cited by 11,505
Authoritative sources in a hyperlinked environment
Kleinberg, J. M. (1999) · Journal of the ACM · doi:10.1145/324133.324140 · Cited by 8,959
Kernel Methods for Pattern Analysis
Shawe-Taylor, J., & Cristianini, N. (2004) · Cambridge University Press eBooks · doi:10.1017/cbo9780511809682 · Cited by 6,587
Cumulated gain-based evaluation of IR techniques
Järvelin, K., & Kekäläinen, J. (2002) · ACM Transactions on Information Systems · doi:10.1145/582415.582418 · Cited by 4,841
Regression Models for Ordinal Data
McCullagh, P. (1980) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1980.tb01109.x · Cited by 4,453
Research works printed but not linked: 24
- Bartell, (1995), Learning to retrieve information
- Bousquet, (2002), Stability and generalization, The Journal of Machine Learning Research, 2, 449
- Burges, (2006), Learning to rank with nonsmooth cost functions, 395
- Carvalho, (2008), A metaleaning approach for robust rank learning
- Chapelle, (2007), Large margin optimization of ranking measures
- Cohen, (1998), Learning to order things, 243
- Diaz, (2007), Regularizing query-based retrieval scores, Information Retrieval, 10, 531, 10.1007/s10791-007-9034-8 · doi:10.1007/s10791-007-9034-8
- Freund, (2003), An efficient boosting algorithm for combining preferences, Journal of Machine Learning Research, 4, 933
- Gyongyi, (2004), Combating web spam with trustrank, 576
- Harrington, (2003), Online ranking/collaborative filtering using the perceptron algorithm, 250
- Herbrich, (2001), Bayes point machines, Journal of Machine Learning Research, 1, 245
- Hersh, (1994), OHSUMED: An interactive retrieval evaluation and new large test collection for research, 192
- Joachims, (2003), Text Mining, 79
- Li, (2003), A loss function analysis for classification methods in text categorization, 472
- Metzler, (2008), Machine learned sentence selection strategies for query-biased summarization
- Mitchell, (1997), Machine Learning
- Pahikkala, (2007), Learning to rank with pairwise regularized least-squares
- Rajaram, (2005), Generalization bounds for k-partite ranking
- Rigutini, (2008), Learning to rank by a neural-based sorting algorithm
- Rocchio, (1971), The SMART Retrieval System — Experiments in Automatic Document Processing, 313
- Shakery, (2006), A probabilistic relevance propagation model for hypertext retrieval, 550
- Tao, (2006), Regularized estimation of mixture models for robust pseudo-relevance feedback, 162, 10.1145/1148170.1148201 · doi:10.1145/1148170.1148201
- Xu, (2005), Ranking definitions with supervised learning methods, 811
- Zhai, (2008), Language models, Foundations and Trends in Information Retrieval
Cited by
Survey on deep learning with class imbalance
Johnson, J. M., & Khoshgoftaar, T. M. (2019) · Journal of Big Data · doi:10.1186/s40537-019-0192-5 · Cited by 3,032
Learning to rank for information retrieval
Liu, T.-Y. (2010) · Proceedings of the 33rd international ACM SIGIR conference on Research and development in information retrieval · doi:10.1145/1835449.1835676 · Cited by 1,856
Self-citationParikh, D., & Grauman, K. (2011) · 2011 International Conference on Computer Vision · doi:10.1109/iccv.2011.6126281 · Cited by 851
Spotting fake reviewer groups in consumer reviews
Mukherjee, A., Liu, B., & Glance, N. (2012) · Proceedings of the 21st international conference on World Wide Web · doi:10.1145/2187836.2187863 · Cited by 773
Entity Linking with a Knowledge Base: Issues, Techniques, and Solutions
Shen, W., Wang, J., & Han, J. (2015) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2014.2327028 · Cited by 761
Credit Card Fraud Detection: A Realistic Modeling and a Novel Learning Strategy
Dal Pozzolo, A., Boracchi, G., Caelen, O., Alippi, C., & Bontempi, G. (2018) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2017.2736643 · Cited by 682
Learning to Navigate for Fine-Grained Classification
Yang, Z., Luo, T., Wang, D., Hu, Z., Gao, J., & Wang, L. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01264-9_26 · Cited by 616
Unbiased Learning-to-Rank with Biased Feedback
Joachims, T., Swaminathan, A., & Schnabel, T. (2017) · Proceedings of the Tenth ACM International Conference on Web Search and Data Mining · doi:10.1145/3018661.3018699 · Cited by 531
Learning to Rank Using User Clicks and Visual Features for Image Retrieval
Yu, J., Tao, D., Wang, M., & Rui, Y. (2015) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2014.2336697 · Cited by 445
Spotting opinion spammers using behavioral footprints
Mukherjee, A., Kumar, A., Liu, B., Wang, J., Hsu, M., Castellanos, M., & Ghosh, R. (2013) · Proceedings of the 19th ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/2487575.2487580 · Cited by 426
Imbalanced Deep Learning by Minority Class Incremental Rectification
Dong, Q., Gong, S., & Zhu, X. (2019) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2018.2832629 · Cited by 369
Deep Multimodal Distance Metric Learning Using Click Constraints for Image Ranking
Yu, J., Yang, X., Gao, F., & Tao, D. (2017) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2016.2591583 · Cited by 364
dipIQ: Blind Image Quality Assessment by Learning-to-Rank Discriminable Image Pairs
Ma, K., Liu, W., Liu, T., Wang, Z., & Tao, D. (2017) · IEEE Transactions on Image Processing · doi:10.1109/tip.2017.2708503 · Cited by 353
Self-citationA Deep Look into neural ranking models for information retrieval
Guo, J., Fan, Y., Pang, L., Yang, L., Ai, Q., Zamani, H., Wu, C., Croft, W. B., & Cheng, X. (2020) · Information Processing & Management · doi:10.1016/j.ipm.2019.102067 · Cited by 349
A Short Introduction to Learning to Rank
LI, H. (2011) · IEICE Transactions on Information and Systems · doi:10.1587/transinf.e94.d.1854 · Cited by 348
DeepFL: integrating multiple fault diagnosis dimensions for deep fault localization
Li, X., Li, W., Zhang, Y., & Zhang, L. (2019) · Proceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis · doi:10.1145/3293882.3330574 · Cited by 336
An Introduction to Neural Information Retrieval
Mitra, B., & Craswell, N. (2018) · Foundations and Trends® in Information Retrieval · doi:10.1561/1500000061 · Cited by 317
Formalizing Visualization Design Knowledge as Constraints: Actionable and Extensible Models in Draco
Moritz, D., Wang, C., Nelson, G. L., Lin, H., Smith, A. M., Howe, B., & Heer, J. (2019) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2018.2865240 · Cited by 317
Rank Pooling for Action Recognition
Fernando, B., Gavves, E., Oramas M., J., Ghodrati, A., & Tuytelaars, T. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2558148 · Cited by 304
Explicit Semantic Ranking for Academic Search via Knowledge Graph Embedding
Xiong, C., Power, R., & Callan, J. (2017) · Proceedings of the 26th International Conference on World Wide Web · doi:10.1145/3038912.3052558 · Cited by 301
Pairwise ranking aggregation in a crowdsourced setting
Chen, X., Bennett, P. N., Collins-Thompson, K., & Horvitz, E. (2013) · Proceedings of the sixth ACM international conference on Web search and data mining · doi:10.1145/2433396.2433420 · Cited by 290
Learning to Rank with Selection Bias in Personal Search
Wang, X., Bendersky, M., Metzler, D., & Najork, M. (2016) · Proceedings of the 39th International ACM SIGIR conference on Research and Development in Information Retrieval · doi:10.1145/2911451.2911537 · Cited by 284
An Empirical Study of Fault Localization Families and Their Combinations
Zou, D., Liang, J., Xiong, Y., Ernst, M. D., & Zhang, L. (2021) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2019.2892102 · Cited by 270
Position Bias Estimation for Unbiased Learning to Rank in Personal Search
Wang, X., Golbandi, N., Bendersky, M., Metzler, D., & Najork, M. (2018) · Proceedings of the Eleventh ACM International Conference on Web Search and Data Mining · doi:10.1145/3159652.3159732 · Cited by 268
More Accurate Question Answering on Freebase
Bast, H., & Haussmann, E. (2015) · Proceedings of the 24th ACM International on Conference on Information and Knowledge Management · doi:10.1145/2806416.2806472 · Cited by 257
Optimizing Dense Retrieval Model Training with Hard Negatives
Zhan, J., Mao, J., Liu, Y., Guo, J., Zhang, M., & Ma, S. (2021) · Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval · doi:10.1145/3404835.3462880 · Cited by 248
Mining query logs: turning search usage data into knowledge
Silvestri, F. (2009) · Foundations and Trends® in Information Retrieval · doi:10.1561/1500000013 · Cited by 194
Best Match: New relevance search for PubMed
Fiorini, N., Canese, K., Starchenko, G., Kireev, E., Kim, W., Miller, V., Osipov, M., Kholodov, M., Ismagilov, R., Mohan, S., et al. (2018) · PLOS Biology · doi:10.1371/journal.pbio.2005343 · Cited by 175
Large Language Models can Accurately Predict Searcher Preferences
Thomas, P., Spielman, S., Craswell, N., & Mitra, B. (2024) · Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval · doi:10.1145/3626772.3657707 · Cited by 168
A General Framework for Counterfactual Learning-to-Rank
Agarwal, A., Takatsu, K., Zaitsev, I., & Joachims, T. (2019) · Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval · doi:10.1145/3331184.3331202 · Cited by 115
Rethinking search: making domain experts out of dilettantes
Metzler, D., Tay, Y., Bahri, D., & Najork, M. (2021) · ACM SIGIR Forum · doi:10.1145/3476415.3476428 · Cited by 112
MeSH Now: Automatic MeSH indexing at PubMed scale via learning to rank
Mao, Y., & Lu, Z. (2017) · Journal of Biomedical Semantics · doi:10.1186/s13326-017-0123-3 · Cited by 87
Turning Clicks into Purchases: Revenue Optimization for Product Search in E-Commerce
Wu, L., Hu, D., Hong, L., & Liu, H. (2018) · The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval · doi:10.1145/3209978.3209993 · Cited by 85
RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMs
Chaudhari, S., Aggarwal, P., Murahari, V., Rajpurohit, T., Kalyan, A., Narasimhan, K., Deshpande, A., & Castro da Silva, B. (2026) · ACM Computing Surveys · doi:10.1145/3743127 · Cited by 92
Chen, C., Zhao, L., Bian, J., Xing, C., & Liu, T.-Y. (2019) · Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining · doi:10.1145/3292500.3330663 · Cited by 73
Self-citationBeyond Ranking: Optimizing Whole-Page Presentation
Wang, Y., Yin, D., Jie, L., Wang, P., Yamada, M., Chang, Y., & Mei, Q. (2016) · Proceedings of the Ninth ACM International Conference on Web Search and Data Mining · doi:10.1145/2835776.2835824 · Cited by 66
Off the Beaten Path: Let's Replace Term-Based Retrieval with k-NN Search
Boytsov, L., Novak, D., Malkov, Y., & Nyberg, E. (2016) · Proceedings of the 25th ACM International on Conference on Information and Knowledge Management · doi:10.1145/2983323.2983815 · Cited by 50
Learning to rank search results for time-sensitive queries
Kanhabua, N., & Nørvåg, K. (2012) · Proceedings of the 21st ACM international conference on Information and knowledge management · doi:10.1145/2396761.2398667 · Cited by 26
Ranking related news predictions
Kanhabua, N., Blanco, R., & Matthews, M. (2011) · Proceedings of the 34th international ACM SIGIR conference on Research and development in Information Retrieval · doi:10.1145/2009916.2010018 · Cited by 24
Conversational AI over Military Scenarios Using Intent Detection and Response Generation
Chuang, H.-M., & Cheng, D.-W. (2022) · Applied Sciences · doi:10.3390/app12052494 · Cited by 15
Kazi, S., Khoja, S., & Daud, A. (2025) · Computer Science Review · doi:10.1016/j.cosrev.2025.100756 · Cited by 9
Li, Y., Fu, X., Verma, G., Buitelaar, P., & Liu, M. (2025) · arXiv · doi:10.48550/arxiv.2510.24476 · Cited by 4
Mind2Web 2: Evaluating Agentic Search with Agent-as-a-Judge
Gou, B., Huang, Z., Ning, Y., Gu, Y., Lin, M., Qi, W., Kopanev, A., Yu, B., Gutiérrez, B. J., Shu, Y., et al. (2025) · arXiv · doi:10.48550/arxiv.2506.21506 · Cited by 3
Fact, Fetch, and Reason: A Unified Evaluation of Retrieval-Augmented Generation
Krishna, S., Krishna, K., Mohananey, A., Schwarcz, S., Stambler, A., Upadhyay, S., & Faruqui, M. (2024) · arXiv · doi:10.48550/arxiv.2409.12941 · Cited by 1
Ai, Q., Zhan, J., & Liu, Y. (2025) · arXiv · doi:10.48550/arxiv.2501.02842
Metadata
- Authors
- Liu, T.-Y.
- Venue
- Foundations and Trends® in Information Retrieval
- Publisher
- Emerald
- Published
- 2009-06-27
- Volume
- 3
- Issue
- 3
- Pages
- 225-331
- ISSN
- 1554-0669 (ISSN-L)
- 1554-0677
- OpenAlex
- W2149427297
- Cited by
- 1,568
- References
- 140
- FWCI
- 133.29
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.