Improving web search ranking by incorporating user behavior information
The DOI matches its Crossref record; 25 of the 25 references it cites were found in a registry or an index.
Abstract
We show that incorporating user behavior data can significantly improve ordering of top results in real web search setting. We examine alternatives for incorporating feedback into the ranking process and explore the contributions of user feedback compared to other common web search features. We report results of a large scale evaluation over 3,000 queries and 12 million user interactions with a popular web search engine. We show that incorporating implicit feedback can augment other features, improving the accuracy of a competitive web search ranking algorithms by as much as 31% relative to the original performance.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The anatomy of a large-scale hypertextual Web search engine
Brin, S., & Page, L. (1998) · Computer Networks and ISDN Systems · doi:10.1016/s0169-7552(98)00110-x · Cited by 16,004
Baeza-Yates, R. A., & Ribeiro-Neto, B. A. (1999) · Cited by 11,505
Introduction to Modern Information Retrieval
Salton, G., & McGill, M. J. (1983) · Virtual Defense Library (Ministerio de Defensa) · Cited by 11,389
Introduction to modern information retrieval
Dillon, M. (1983) · Information Processing & Management · doi:10.1016/0306-4573(83)90062-6 · Cited by 5,909
Making Large-Scale SVM Learning Practical
Joachims, T. (1998) · RePEc: Research Papers in Economics · doi:10.17877/de290r-14262 · Cited by 4,411
Optimizing search engines using clickthrough data
Joachims, T. (2002) · Proceedings of the eighth ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/775047.775067 · Cited by 3,873
Learning to rank using gradient descent
Burges, C., Shaked, T., Renshaw, E., Lazier, A., Deeds, M., Hamilton, N., & Hullender, G. (2005) · Proceedings of the 22nd international conference on Machine learning - ICML '05 · doi:10.1145/1102351.1102363 · Cited by 2,836 · cited 16×
“We use a 2-layer implementation of RankNet in order to model non-linear relationships between features.”
InfluentialGroupLens: applying collaborative filtering to Usenet news
Konstan, J. A., Miller, B. N., Maltz, D., Herlocker, J. L., Gordon, L. R., & Riedl, J. (1997) · Communications of the ACM · doi:10.1145/245108.245126 · Cited by 2,367
IR evaluation methods for retrieving highly relevant documents
Järvelin, K., & Kekäläinen, J. (2017) · ACM SIGIR Forum · doi:10.1145/3130348.3130374 · Cited by 1,444
Accurately interpreting clickthrough data as implicit feedback
Joachims, T., Granka, L., Pan, B., Hembrooke, H., & Gay, G. (2005) · Proceedings of the 28th annual international ACM SIGIR conference on Research and development in information retrieval · doi:10.1145/1076034.1076063 · Cited by 1,408
IR evaluation methods for retrieving highly relevant documents
Järvelin, K., & Kekäläinen, J. (2000) · Proceedings of the 23rd annual international ACM SIGIR conference on Research and development in information retrieval · doi:10.1145/345508.345545 · Cited by 967
Implicit feedback for inferring user preference: a bibliography
Kelly, D., & Teevan, J. (2003) · ACM SIGIR Forum · doi:10.1145/959258.959260 · Cited by 743
Simple BM25 extension to multiple weighted fields
Robertson, S., Zaragoza, H., & Taylor, M. (2004) · Proceedings of the thirteenth ACM international conference on Information and knowledge management · doi:10.1145/1031171.1031181 · Cited by 701
Schölkopf, B. (1997) · Cited by 559
Query chains: learning to rank from implicit feedback
Radlinski, F., & Joachims, T. (2005) · Proceedings of the eleventh ACM SIGKDD international conference on Knowledge discovery in data mining · doi:10.1145/1081870.1081899 · Cited by 476
Learning user interaction models for predicting web search result preferences
Agichtein, E., Brill, E., Dumais, S., & Ragno, R. (2006) · Proceedings of the 29th annual international ACM SIGIR conference on Research and development in information retrieval · doi:10.1145/1148170.1148175 · Cited by 462
Self-citationEvaluating implicit measures to improve web search
Fox, S., Karnawat, K., Mydland, M., Dumais, S., & White, T. (2005) · ACM Transactions on Information Systems · doi:10.1145/1059981.1059982 · Cited by 412
Self-citationImplicit Feedback for Recommender Systems
Oard, D. W., & Kim, J. (1998) · Scholar Commons (University of South Carolina) · Cited by 371
Optimizing web search using web click-through data
Xue, G.-R., Zeng, H.-J., Chen, Z., Yu, Y., Ma, W.-Y., Xi, W., & Fan, W. (2004) · Proceedings of the thirteenth ACM international conference on Information and knowledge management · doi:10.1145/1031171.1031192 · Cited by 249
Learning users' interests by unobtrusively observing their normal behavior
Goecks, J., & Shavlik, J. (2000) · Proceedings of the 5th international conference on Intelligent user interfaces · doi:10.1145/325737.325806 · Cited by 182
Claypool, M. K., Brown, D. C., Le, P., & Waseda, M. (2001) · IEEE Internet Computing · doi:10.1109/4236.968829 · Cited by 169
Modeling Information Content Using Observable Behavior
Oard, D. W., & Kim, J. (2001) · Scholar Commons (University of South Carolina) · Cited by 143
The SST method: a tool for analysing Web information search processes
Pharo, N., & Järvelin, K. (2003) · Information Processing & Management · doi:10.1016/j.ipm.2003.08.005 · Cited by 61
The Use of Proximal Information Scent to Forage for Distal Content on the World Wide Web
Pirolli, P. L. T. (2009) · Oxford University Press eBooks · doi:10.1093/acprof:oso/9780195374827.003.0022 · Cited by 45
Evaluating the Robustness of Learning from Implicit FeedbackPreprint
Radlinski, F., & Joachims, T. (2006) · arXiv (Cornell University) · doi:10.48550/arxiv.cs/0605036 · Cited by 23
Cited by
Filter Bubbles, Echo Chambers, and Online News Consumption
Flaxman, S., Goel, S., & Rao, J. M. (2016) · Public Opinion Quarterly · doi:10.1093/poq/nfw006 · Cited by 1,992
Learning to rank for information retrieval
Liu, T.-Y. (2010) · Proceedings of the 33rd international ACM SIGIR conference on Research and development in information retrieval · doi:10.1145/1835449.1835676 · Cited by 1,856
Predicting clicks: estimating the click-through rate for new ads
Richardson, M., Dominowska, E., & Ragno, R. (2007) · Proceedings of the 16th international conference on World Wide Web · doi:10.1145/1242572.1242643 · Cited by 1,049
Novelty and diversity in information retrieval evaluation
Clarke, C. L. A., Kolla, M., Cormack, G. V., Vechtomova, O., Ashkan, A., Büttcher, S., & MacKinnon, I. (2008) · Proceedings of the 31st annual international ACM SIGIR conference on Research and development in information retrieval · doi:10.1145/1390334.1390446 · Cited by 962
Evaluating the accuracy of implicit feedback from clicks and query reformulations in Web search
Joachims, T., Granka, L., Pan, B., Hembrooke, H., Radlinski, F., & Gay, G. (2007) · ACM Transactions on Information Systems · doi:10.1145/1229179.1229181 · Cited by 677
Metadata
- Authors
- Agichtein, E., Brill, E., & Dumais, S.
- Venue
- Proceedings of the 29th annual international ACM SIGIR conference on Research and development in information retrieval
- Publisher
- ACM
- Published
- 2006-08-06
- Pages
- 19-26
- OpenAlex
- W2125771191
- Cited by
- 1,041
- References
- 25
- FWCI
- 50.01
- Type
- conference paper
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.