Evaluating the accuracy of implicit feedback from clicks and query reformulations in Web search
The DOI matches its Crossref record; 48 of the 48 references it cites were found in a registry or an index.
Abstract
This article examines the reliability of implicit feedback generated from clickthrough data and query reformulations in World Wide Web (WWW) search. Analyzing the users' decision process using eyetracking and comparing implicit feedback against manual relevance judgments, we conclude that clicks are informative but biased. While this makes the interpretation of clicks as absolute relevance judgments difficult, we show that relative preferences derived from clicks are reasonably accurate on average. We find that such relative preferences are accurate not only between results from an individual query, but across multiple sets of results within chains of query reformulations.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Eye movements in reading and information processing: 20 years of research
Rayner, K. (1998) · Psychological Bulletin · doi:10.1037/0033-2909.124.3.372 · Cited by 7,979
A theory of reading: From eye fixations to comprehension.
Just, M. A., & Carpenter, P. A. (1980) · Psychological Review · doi:10.1037/0033-295x.87.4.329 · Cited by 5,220
Optimizing search engines using clickthrough data
Joachims, T. (2002) · Proceedings of the eighth ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/775047.775067 · Cited by 3,873
Self-citationLearning to rank using gradient descent
Burges, C., Shaked, T., Renshaw, E., Lazier, A., Deeds, M., Hamilton, N., & Hullender, G. (2005) · Proceedings of the 22nd international conference on Machine learning - ICML '05 · doi:10.1145/1102351.1102363 · Cited by 2,836
Varian, H. R. (1992) · Cited by 2,080
Broder, A. (2002) · ACM SIGIR Forum · doi:10.1145/792550.792552 · Cited by 1,944 · cited 4×
“The other five tasks were informational [Broder 2002], asking subjects to find a specific bit of information.”
BackgroundMethodInfluentialAccurately interpreting clickthrough data as implicit feedback
Joachims, T., Granka, L., Pan, B., Hembrooke, H., & Gay, G. (2005) · Proceedings of the 28th annual international ACM SIGIR conference on Research and development in information retrieval · doi:10.1145/1076034.1076063 · Cited by 1,408
Self-citationReal life, real users, and real needs: a study and analysis of user queries on the web
Jansen, B. J., Spink, A., & Saracevic, T. (2000) · Information Processing & Management · doi:10.1016/s0306-4573(99)00056-4 · Cited by 1,320
Needham, D., & Lloyd, C. (1968) · Economica · doi:10.2307/2552169 · Cited by 1,278
Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI 2017)
Amendola, G., Ricca, F., & Truszczyński, M. (2017) · International Joint Conference on Artificial Intelligence · Cited by 1,179
Analysis of a very large web search engine query log
Silverstein, C., Marais, H., Henzinger, M., & Moricz, M. (1999) · ACM SIGIR Forum · doi:10.1145/331403.331405 · Cited by 1,153
Letizia: an agent that assists web browsing
Lieberman, H. A. (1995) · Cited by 1,137
Eye movements in reading and information processing.
Rayner, K. (1978) · Psychological Bulletin · doi:10.1037//0033-2909.85.3.618 · Cited by 1,119
Large Margin Rank Boundaries for Ordinal Regression
Herbrich, R., Graepel, T., & Obermayer, K. (2000) · Advances in Large-Margin Classifiers · doi:10.7551/mitpress/1113.003.0010 · Cited by 1,074
Improving web search ranking by incorporating user behavior information
Agichtein, E., Brill, E., & Dumais, S. (2006) · Proceedings of the 29th annual international ACM SIGIR conference on Research and development in information retrieval · doi:10.1145/1148170.1148177 · Cited by 1,041
Consumption Theory in Terms of Revealed Preference
Samuelson, P. A. (1948) · Economica · doi:10.2307/2549561 · Cited by 984
Implicit feedback for inferring user preference: a bibliography
Kelly, D., & Teevan, J. (2003) · ACM SIGIR Forum · doi:10.1145/959258.959260 · Cited by 743
Eye-tracking analysis of user behavior in WWW search
Granka, L. A., Joachims, T., & Gay, G. (2004) · Proceedings of the 27th annual international ACM SIGIR conference on Research and development in information retrieval · doi:10.1145/1008992.1009079 · Cited by 696
Self-citationWeb Watcher: A Tour Guide for the World Wide Web.
Joachims, T., Freitag, D., & Mitchell, T. M. (1997) · Cited by 685
Self-citationClaypool, M., Le, P., Wased, M., & Brown, D. (2001) · Proceedings of the 6th international conference on Intelligent user interfaces · doi:10.1145/359784.359836 · Cited by 666
Crammer, K., & Singer, Y. (2002) · Advances in Neural Information Processing Systems 14 · doi:10.7551/mitpress/1120.003.0087 · Cited by 538
Query chains: learning to rank from implicit feedback
Radlinski, F., & Joachims, T. (2005) · Proceedings of the eleventh ACM SIGKDD international conference on Knowledge discovery in data mining · doi:10.1145/1081870.1081899 · Cited by 476
Self-citationLearning user interaction models for predicting web search result preferences
Agichtein, E., Brill, E., Dumais, S., & Ragno, R. (2006) · Proceedings of the 29th annual international ACM SIGIR conference on Research and development in information retrieval · doi:10.1145/1148170.1148175 · Cited by 462
Context-sensitive information retrieval using implicit feedback
Shen, X., Tan, B., & Zhai, C. (2005) · Proceedings of the 28th annual international ACM SIGIR conference on Research and development in information retrieval · doi:10.1145/1076034.1076045 · Cited by 436
Evaluating implicit measures to improve web search
Fox, S., Karnawat, K., Mydland, M., Dumais, S., & White, T. (2005) · ACM Transactions on Information Systems · doi:10.1145/1059981.1059982 · Cited by 412
Eye tracking in web search tasks: design implications
Goldberg, J. H., Stimson, M. J., Lewenstein, M., Scott, N., & Wichansky, A. M. (2002) · Proceedings of the symposium on Eye tracking research & applications - ETRA '02 · doi:10.1145/507072.507082 · Cited by 382
Implicit Feedback for Recommender Systems
Oard, D. W., & Kim, J. (1998) · Scholar Commons (University of South Carolina) · Cited by 371
Cohen, W. W., Schapire, R. E., & Singer, Y. (1999) · Journal of Artificial Intelligence Research · doi:10.1613/jair.587 · Cited by 317
The determinants of web page viewing behavior: an eye-tracking study
Pan, B., Hembrooke, H. A., Gay, G. K., Granka, L. A., Feusner, M. K., & Newman, J. K. (2004) · Proceedings of the Eye tracking research & applications symposium on Eye tracking research & applications - ETRA'2004 · doi:10.1145/968363.968391 · Cited by 287
Self-citationMantel, S. P., & Kardes, F. R. (1999) · Journal of Consumer Research · doi:10.1086/209543 · Cited by 280
Automatic Combination of Multiple Ranked Retrieval Systems
Bartell, B. T., Cottrell, G. W., & Belew, R. K. (1994) · SIGIR ’94 · doi:10.1007/978-1-4471-2099-5_18 · Cited by 277
Display time as implicit feedback: understanding task effects
Kelly, D., & Belkin, N. J. (2004) · Proceedings of the 27th annual international ACM SIGIR conference on Research and development in information retrieval · doi:10.1145/1008992.1009057 · Cited by 228
Ignoring Irrelevant Information: Situational Determinants of Consumer Learning
Hutchinson, J. W., & Alba, J. W. (1991) · Journal of Consumer Research · doi:10.1086/209263 · Cited by 197
Optimum polynomial retrieval functions based on the probability ranking principle
Fuhr, N. (1989) · ACM Transactions on Information Systems · doi:10.1145/65943.65944 · Cited by 138
A Machine Learning Architecture for Optimizing Web Search Engines
Boyan, J. A., & Freitag, D. (1999) · Cited by 130
The Use of Implicit Evidence for Relevance Feedback in Web Retrieval
White, R. W., Ruthven, I. G., & Jose, J. M. (2002) · Lecture notes in computer science · doi:10.1007/3-540-45886-7_7 · Cited by 123
A study of factors affecting the utility of implicit relevance feedback
White, R. W., Ruthven, I. G., & Jose, J. M. (2005) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/1076034.1076044 · Cited by 120
Preference Mining: A Novel Approach on Mining User Preferences for Personalized Applications
Holland, S., Ester, M., & Kießling, W. (2003) · Lecture notes in computer science · doi:10.1007/978-3-540-39804-2_20 · Cited by 108
Depth- and breadth-first processing of search result lists
Klöckner, K., Wirschum, N., & Jameson, A. (2004) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/985921.986115 · Cited by 83
Improving Web Search Ranking by Incorporating User Behavior
Agichtein, E., Brill, E., & Dumais, S. (2006) · Cited by 76
Beyond the Commons: Investigating the Value of Personalizing Web Search
Teevan, J. B., Dumais, S., & Horvitz, E. (2005) · Cited by 75
Gazetracker: software designed to facilitate eye movement analysis
Lankford, C. (2000) · Proceedings of the symposium on Eye tracking research & applications - ETRA '00 · doi:10.1145/355017.355025 · Cited by 70
A community-aware search engine
Almeida, R. B., & Almeida, V. A. F. (2004) · International World Wide Web Conference (WWW) · doi:10.1145/988672.988728 · Cited by 70
Experience with an adaptive indexing scheme
Furnas, G. W. (1985) · ACM SIGCHI Bulletin · doi:10.1145/1165385.317480 · Cited by 61
Can Relevance be Inferred from Eye Movements in Information Retrieval
Salojärvi, J., Kojo, I., Simola, J., & Samuel Kaski, A. (2003) · Cited by 47
Good enough but I'll just check: Web-page search as attentional refocusing
Brumby, D. P., & Howes, A. (2004) · Cited by 43
Long-Term Learning for Web Search Engines
Kemp, C., & Ramamohanarao, K. (2002) · Lecture notes in computer science · doi:10.1007/3-540-45681-3_22 · Cited by 41
Halverson, T., & Hornof, A. J. (2004) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/985921.986066 · Cited by 19
Cited by
A Survey of Automatic Query Expansion in Information Retrieval
Carpineto, C., & Romano, G. (2012) · ACM Computing Surveys · doi:10.1145/2071389.2071390 · Cited by 1,109
Bias and Debias in Recommender System: A Survey and Future Directions
Chen, J., Dong, H., Wang, X., Feng, F., Wang, M., & He, X. (2023) · ACM Transactions on Information Systems · doi:10.1145/3564284 · Cited by 741
“For example, recent studies on eye tracking demonstrate that users are less likely to browse items that are ranked lower in vertical lists, while they only examine the first few items at the top of lists [34], [35].”
BackgroundThe search engine manipulation effect (SEME) and its possible impact on the outcomes of elections
Epstein, R., & Robertson, R. E. (2015) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1419828112 · Cited by 577
Click Prediction for Web Image Reranking Using Multimodal Sparse Coding
Yu, J., Rui, Y., & Tao, D. (2014) · IEEE Transactions on Image Processing · doi:10.1109/tip.2014.2311377 · Cited by 550
Unbiased Learning-to-Rank with Biased Feedback
Joachims, T., Swaminathan, A., & Schnabel, T. (2017) · Proceedings of the Tenth ACM International Conference on Web Search and Data Mining · doi:10.1145/3018661.3018699 · Cited by 531
“Second, treating clicks as preferences between clicked and skipped documents has been found to be accurate [9, 11], but it can only infer preferences that oppose the presented order.”
BackgroundInfluentialSelf-citation
Metadata
- Authors
- Joachims, T., Granka, L., Pan, B., Hembrooke, H., Radlinski, F., & Gay, G.
- Venue
- ACM Transactions on Information Systems
- Published
- 2007-04
- Volume
- 25
- Issue
- 2
- Pages
- 7
- ISSN
- 1046-8188 (ISSN-L)
- 1558-2868
- OpenAlex
- W1974360117
- Cited by
- 677
- References
- 48
- FWCI
- 102.05
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.