The DOI matches its Crossref record; 23 of the 23 references it cites were found in a registry or an index.
How does clickthrough data reflect retrieval quality?
Abstract
Automatically judging the quality of retrieval functions based on observable user behavior holds promise for making retrieval evaluation faster, cheaper, and more user centered. However, the relationship between observable user behavior and retrieval quality is not yet fully understood. We present a sequence of studies investigating this relationship for an operational search engine on the arXiv.org e-print archive. We find that none of the eight absolute usage metrics we explore (e.g., number of clicks, frequency of query reformulations, abandonment) reliably reflect retrieval quality for the sample sizes we consider. However, we find that paired experiment designs adapted from sensory analysis produce accurate and reliable statements about the relative quality of two retrieval functions. In particular, we investigate two paired comparison tests that analyze clickthrough data from an interleaved presentation of ranking pairs, and we find that both give accurate and consistent results. We conclude that both paired comparison tests give substantially more accurate and sensitive evaluation results than absolute usage metrics in our domain.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Introduction to information retrieval
(2009) · Choice Reviews Online · doi:10.5860/choice.46-2715 · Cited by 12,278
Optimizing search engines using clickthrough data
Joachims, T. (2002) · Proceedings of the eighth ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/775047.775067 · Cited by 3,873
Self-citationTREC: Experiment and evaluation in information retrieval
Vicedo, J. L., & Gómez, J. (2007) · Journal of the American Society for Information Science and Technology · doi:10.1002/asi.20583 · Cited by 837
Implicit feedback for inferring user preference: a bibliography
Kelly, D., & Teevan, J. (2003) · ACM SIGIR Forum · doi:10.1145/959258.959260 · Cited by 743
Retrieval evaluation with incomplete information
Buckley, C., & Voorhees, E. M. (2004) · Proceedings of the 27th annual international ACM SIGIR conference on Research and development in information retrieval · doi:10.1145/1008992.1009000 · Cited by 736
Evaluating the accuracy of implicit feedback from clicks and query reformulations in Web search
Joachims, T., Granka, L., Pan, B., Hembrooke, H., Radlinski, F., & Gay, G. (2007) · ACM Transactions on Information Systems · doi:10.1145/1229179.1229181 · Cited by 677
Self-citationLearning user interaction models for predicting web search result preferences
Agichtein, E., Brill, E., Dumais, S., & Ragno, R. (2006) · Proceedings of the 29th annual international ACM SIGIR conference on Research and development in information retrieval · doi:10.1145/1148170.1148175 · Cited by 462
Evaluating implicit measures to improve web search
Fox, S., Karnawat, K., Mydland, M., Dumais, S., & White, T. (2005) · ACM Transactions on Information Systems · doi:10.1145/1059981.1059982 · Cited by 412
Voorhees, E. M., & Harman, D. K. (2005) · The MIT Press eBooks · Cited by 392
User performance versus precision measures for simple search tasks
Turpin, A., & Scholer, F. (2006) · Proceedings of the 29th annual international ACM SIGIR conference on Research and development in information retrieval · doi:10.1145/1148170.1148176 · Cited by 389
Cited by
Expected reciprocal rank for graded relevance
Chapelle, O., Metlzer, D., Zhang, Y., & Grinspan, P. (2009) · Proceedings of the 18th ACM conference on Information and knowledge management · doi:10.1145/1645953.1646033 · Cited by 832
Test Collection Based Evaluation of Information Retrieval Systems
Sanderson, M. (2010) · Foundations and Trends® in Information Retrieval · doi:10.1561/1500000009 · Cited by 485
Chuklin, A., Markov, I., & de Rijke, M. (2015) · Synthesis Lectures on Information Concepts, Retrieval, and Services · doi:10.1007/978-3-031-02294-4 · Cited by 398
Interactively optimizing information retrieval systems as a dueling bandits problem
Yue, Y., & Joachims, T. (2009) · Proceedings of the 26th Annual International Conference on Machine Learning · doi:10.1145/1553374.1553527 · Cited by 268
Self-citationUnderstanding PubMed(R) user search behavior through log analysis
Islamaj Dogan, R., Murray, G. C., Neveol, A., & Lu, Z. (2009) · Database · doi:10.1093/database/bap018 · Cited by 265
Metadata
- Authors
- Radlinski, F., Kurup, M., & Joachims, T.
- Venue
- Proceedings of the 17th ACM conference on Information and knowledge management
- Publisher
- ACM
- Published
- 2008-10-26
- Pages
- 43-52
- OpenAlex
- W2094790959
- Cited by
- 293
- References
- 23
- FWCI
- 15.21
- Type
- conference paper
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: How does clickthrough data reflect retrieval quality? https://app.cochat.ai/papers/10.1145/1458082.1458092
- The paper
- Radlinski, F., Kurup, M., & Joachims, T. (2008). How does clickthrough data reflect retrieval quality? Proceedings of the 17th ACM conference on Information and knowledge management. https://doi.org/10.1145/1458082.1458092
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.