The DOI matches its Crossref record.
Evaluating implicit measures to improve web search
Abstract
Of growing interest in the area of improving the search experience is the collection of implicit user behavior measures (implicit measures) as indications of user interest and user satisfaction. Rather than having to submit explicit user feedback, which can be costly in time and resources and alter the pattern of use within the search experience, some research has explored the collection of implicit measures as an efficient and useful alternative to collecting explicit measure of interest from users.This research article describes a recent study with two main objectives. The first was to test whether there is an association between explicit ratings of user satisfaction and implicit measures of user interest. The second was to understand what implicit measures were most strongly associated with user satisfaction. The domain of interest was Web search. We developed an instrumented browser to collect a variety of measures of user activity and also to ask for explicit judgments of the relevance of individual pages visited and entire search sessions. The data was collected in a workplace setting to improve the generalizability of the results.Results were analyzed using traditional methods (e.g., Bayesian modeling and decision trees) as well as a new usage behavior pattern analysis (“gene analysis”). We found that there was an association between implicit measures of user activity and the user's explicit satisfaction ratings. The best models for individual pages combined clickthrough, time spent on the search result page, and how a user exited a result or ended a search session (exit type/end action). Behavioral patterns (through the gene analysis) can also be used to predict user satisfaction for search sessions.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Following prior work establishing implicit behaviors as reliable satisfaction indicators (Joachims et al., 2005;Fox et al., 2005;Kim et al., 2014), we use click-through rate (CTR) as our primary success surrogate, as S2 link clicks strongly predict user returns (Appendix D).”
From Understanding Usage and Engagement in AI-Powered Scientific Research Tools: The Asta Interaction Dataset (2026) · cites this paper as Fox et al., 2005
Citations
Cites
No references could be resolved yet.
Research works printed but not linked: 15
- Chickering , D. M. , Heckerman , D. , and Meek , C . 1997. A Bayesian approach to learning bayesian networks with local structure . In Proceedings of the Thirteenth Conference on Uncertainty in Artificial Intelligence. 80--89 . Chickering, D. M., Heckerman, D., and Meek, C. 1997. A Bayesian approach to learning bayesian networks with local structure. In Proceedings of the Thirteenth Conference on Uncertainty in Artificial Intelligence. 80--89.
- 10.1109/4236.968829 · DOI
- 10.1023/A:1022649401552 · DOI
- Goecks , J. and Shavlik , J . 1999. Learning users' interests by unobtrusively observing their normal behavior . In Proceedings of the IJCAI Workshop on Machine Learning for Information Filtering. 129--132 . 10.1145/325737.325806 Goecks, J. and Shavlik, J. 1999. Learning users' interests by unobtrusively observing their normal behavior. In Proceedings of the IJCAI Workshop on Machine Learning for Information Filtering. 129--132. 10.1145/325737.325806 · DOI
- Heckerman D. Geiger D. and Chickering D. M. 1995. Learning Bayesian networks: The combination of knowledge and statistical data. Mach. Learn. 20. 197--243. 10.1023/A:1022623210503 Heckerman D. Geiger D. and Chickering D. M. 1995. Learning Bayesian networks: The combination of knowledge and statistical data. Mach. Learn. 20. 197--243. 10.1023/A:1022623210503 · DOI
- Horvitz , E. , Breese , J. , Heckerman , D. , Hovel , D. , and Rommelse , K . 1998. The Lumiere Project: Bayesian user modeling for inferring the goals and needs of software users . In Proceedings of the Fourteenth Conference on Uncertainty in Artificial Intelligence (July). 256--265 . Horvitz, E., Breese, J., Heckerman, D., Hovel, D., and Rommelse, K. 1998. The Lumiere Project: Bayesian user modeling for inferring the goals and needs of software users. In Proceedings of the Fourteenth Conference on Uncertainty in Artificial Intelligence (July). 256--265.
- Joachims , T. 2002 . Optimizing search engines using clickthrough data . In Proceedings of KDD 2004. 133--142. 10.1145/775047.775067 Joachims, T. 2002. Optimizing search engines using clickthrough data. In Proceedings of KDD 2004. 133--142. 10.1145/775047.775067 · DOI
- 10.1145/959258.959260 · DOI
- 10.1145/245108.245126 · DOI
- Morita , M. and Shinoda , Y . 1994. Information filtering based on user behavior analysis and best match text retrieval . In Proceedings of the Seventeenth Annual International ACM-SIGIR Conference on Research and Development in Information Retrieval (July). 272--281 . Morita, M. and Shinoda, Y. 1994. Information filtering based on user behavior analysis and best match text retrieval. In Proceedings of the Seventeenth Annual International ACM-SIGIR Conference on Research and Development in Information Retrieval (July). 272--281.
- Nichols , D. M. 1997 . Implicit ratings and filtering . In Proceedings of the Fifth DELOS Workshop on Filtering and Collaborative Filtering (Nov.). 221--228 . Nichols, D. M. 1997. Implicit ratings and filtering. In Proceedings of the Fifth DELOS Workshop on Filtering and Collaborative Filtering (Nov.). 221--228.
- Oard , D. and Kim , J . 1998. Implicit feedback for recommender systems . In Proceedings of the AAAI Workshop on Recommender Systems (July). 81--83 . Oard, D. and Kim, J. 1998. Implicit feedback for recommender systems. In Proceedings of the AAAI Workshop on Recommender Systems (July). 81--83.
- Oard , D. W. and Kim , J . 2001. Modeling information content using observable behavior . In Proceedings of the 64th Annual Meeting of the American Society for Information Science and Technology. 38--45 . Oard, D. W. and Kim, J. 2001. Modeling information content using observable behavior. In Proceedings of the 64th Annual Meeting of the American Society for Information Science and Technology. 38--45.
- 10.1002/1097-4571(2000)9999:9999%3C::AID-ASI1591%3E3.3.CO;2-I · DOI
- Voorhees , E. 2001 . Evaluation by highly relevant documents . In Proceedings of the Twenty-Fourth Annual International ACM-SIGIR Conference on Research and Development in Information Retrieval. 74--82 . 10.1145/383952.383963 Voorhees, E. 2001. Evaluation by highly relevant documents. In Proceedings of the Twenty-Fourth Annual International ACM-SIGIR Conference on Research and Development in Information Retrieval. 74--82. 10.1145/383952.383963 · DOI
Cited by
Sorted by citations
Lalmas, M., O’Brien, H., & Yom-Tov, E. (2015) · Synthesis Lectures on Information Concepts, Retrieval, and Services · doi:10.1007/978-3-031-02289-0 · Cited by 192
Predicting User Satisfaction with Intelligent Assistants
Kiseleva, J., Williams, K., Hassan Awadallah, A., Crook, A. C., Zitouni, I., & Anastasakos, T. (2016) · Proceedings of the 39th International ACM SIGIR conference on Research and Development in Information Retrieval · doi:10.1145/2911451.2911521 · Cited by 108
Captions and biases in diagnostic search
White, R. W., & Horvitz, E. (2013) · ACM Transactions on the Web · doi:10.1145/2486040 · Cited by 29
Haddad, D., Bareket, D., Chang, J. C., DeYoung, J., Hwang, J. D., Katz, U., Polak, M., Suh, S., Surana, H., Tiktinsky, A., et al. (2026) · arXiv · doi:10.48550/arxiv.2602.23335 · Cited by 4
Metadata
- Authors
- Fox, S., Karnawat, K., Mydland, M., Dumais, S., & White, T.
- Affiliations
- Microsoft Corp., Redmond, WA
- Venue
- ACM Transactions on Information Systems
- Published
- 2005-04
- Volume
- 23
- Issue
- 2
- Pages
- 147-168
- ISSN
- 1046-8188
- 1558-2868
- Cited by
- 412
- References
- 0
- Language
- en
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Evaluating implicit measures to improve web search. https://app.cochat.ai/papers/10.1145/1059981.1059982
- The paper
- Fox, S., Karnawat, K., Mydland, M., Dumais, S., & White, T. (2005). Evaluating implicit measures to improve web search. ACM Transactions on Information Systems. https://doi.org/10.1145/1059981.1059982
