The DOI matches its Crossref record; 47 of the 48 references it cites were found in a registry or an index.
Struggling and Success in Web Search
Open accessAbstract
Web searchers sometimes struggle to find relevant information. Struggling leads to frustrating and dissatisfying search experiences, even if searchers ultimately meet their search objectives. Better understanding of search tasks where people struggle is important in improving search systems. We address this important issue using a mixed methods study using large-scale logs, crowd-sourced labeling, and predictive modeling. We analyze anonymized search logs from the Microsoft Bing Web search engine to characterize aspects of struggling searches and better explain the relationship between struggling and search success. To broaden our understanding of the struggling process beyond the behavioral signals in log data, we develop and utilize a crowd-sourced labeling methodology. We collect third-party judgments about why searchers appear to struggle and, if appropriate, where in the search task it became clear to the judges that searches would succeed (i.e., the pivotal query). We use our findings to propose ways in which systems can help searchers reduce struggling. Key components of such support are algorithms that accurately predict the nature of future actions and their anticipated impact on search outcomes. Our findings have implications for the design of search systems that help searchers struggle less and succeed more.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Citations
Cites
Classification and Regression Trees.
Gordon, A. D., Breiman, L., Friedman, J. H., Olshen, R. A., & Stone, C. J. (1984) · Biometrics · doi:10.2307/2530946 · Cited by 23,522
Classification and Regression Trees.
Ryzin, J. V., Breiman, L., Friedman, J. H., Olshen, R. A., & Stone, C. J. (1986) · Journal of the American Statistical Association · doi:10.2307/2288003 · Cited by 20,825
Classification And Regression Trees
Breiman, L., Friedman, J. H., Olshen, R. A., & Stone, C. J. (2017) · doi:10.1201/9781315139470 · Cited by 11,417
Satisfaction: a behavioral perspective on the consumer
(2010) · Choice Reviews Online · doi:10.5860/choice.47-6963 · Cited by 6,042
The Oxford Handbook of Positive Psychology
(2009) · doi:10.1093/oxfordhb/9780195187243.001.0001 · Cited by 2,696 · cited 1×
“Satisfaction has been studied extensively in a number of areas such as psychology [30] and commerce [33].”
BackgroundInter-Coder Agreement for Computational Linguistics
Artstein, R., & Poesio, M. (2008) · Computational Linguistics · doi:10.1162/coli.07-034-r2 · Cited by 1,665
Exploratory search: from finding to understanding
Marchionini, G. (2006) · Communications of the ACM · doi:10.1145/1121949.1121979 · Cited by 1,506
Methods for Evaluating Interactive Information Retrieval Systems with Users
Kelly, D. (2009) · Foundations and Trends® in Information Retrieval · doi:10.1561/1500000012 · Cited by 633
Generating query substitutions
Jones, R., Rey, B., Madani, O., & Greiner, W. (2006) · Proceedings of the 15th international conference on World Wide Web · doi:10.1145/1135777.1135835 · Cited by 618
Evaluating implicit measures to improve web search
Fox, S., Karnawat, K., Mydland, M., Dumais, S., & White, T. (2005) · ACM Transactions on Information Systems · doi:10.1145/1059981.1059982 · Cited by 412
Self-citationCharacterizing the influence of domain expertise on web search behavior
White, R. W., Dumais, S. T., & Teevan, J. (2009) · Proceedings of the Second ACM International Conference on Web Search and Data Mining · doi:10.1145/1498759.1498819 · Cited by 312
Self-citationAnalyzing and evaluating query reformulation strategies in web search logs
Huang, J., & Efthimiadis, E. N. (2009) · Proceedings of the 18th ACM conference on Information and knowledge management · doi:10.1145/1645953.1645966 · Cited by 277
Patterns of Search: Analyzing and Modeling Web Query Refinement
Lau, T., & Horvitz, E. (1999) · CISM International Centre for Mechanical Sciences · doi:10.1007/978-3-7091-2490-1_12 · Cited by 253
Using terminological feedback for web search refinement: a log-based study
Anick, P. (2003) · Proceedings of the 26th annual international ACM SIGIR conference on Research and development in informaion retrieval · doi:10.1145/860435.860453 · Cited by 249
Oxford handbook of positive psychology
(2009) · Choice Reviews Online · doi:10.5860/choice.47-1719 · Cited by 221
Beyond DCG: user behavior as a predictor of a successful search
Hassan, A., Jones, R., & Klinkner, K. L. (2010) · Proceedings of the third ACM international conference on Web search and data mining · doi:10.1145/1718487.1718515 · Cited by 213
How does search behavior change as search becomes more difficult?
Aula, A., Khan, R. M., & Guan, Z. (2010) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1753326.1753333 · Cited by 209
Studying the use of popular destinations to enhance web search interaction
White, R. W., Bilenko, M., & Cucerzan, S. (2007) · Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval · doi:10.1145/1277741.1277771 · Cited by 197
Self-citationLessons from the journey: a query log analysis of within-session learning
Eickhoff, C., Teevan, J. B., White, R. W., & Dumais, S. (2014) · ACM International Conference on Web Search and Data Mining (WSDM) · doi:10.1145/2556195.2556217 · Cited by 177
Self-citationInvestigating the querying and browsing behavior of advanced search engine users
White, R. W., & Morris, D. (2007) · Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval · doi:10.1145/1277741.1277787 · Cited by 171
Self-citationPredicting searcher frustration
Feild, H. A., Allan, J., & Jones, R. (2010) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/1835449.1835458 · Cited by 164
Modeling dwell time to predict click-level satisfaction
Kim, Y., Hassan, A., White, R. W., & Zitouni, I. (2014) · Proceedings of the 7th ACM international conference on Web search and data mining · doi:10.1145/2556195.2556220 · Cited by 164
Self-citationHow well does result relevance predict session satisfaction?
Huffman, S. B., & Hochster, M. (2007) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/1277741.1277839 · Cited by 158
Understanding the relationship between searchers' queries and information goals
Downey, D., Dumais, S., Liebling, D. J., & Horvitz, E. (2008) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/1458082.1458143 · Cited by 157
Self-citationClassification-enhanced ranking
Bennett, P. N., Svore, K. M., & Dumais, S. (2010) · International World Wide Web Conference (WWW) · doi:10.1145/1772690.1772703 · Cited by 121
Self-citationСергеевич, А. М., Guo, Q., Lagun, D., & Agichtein, E. (2011) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/2009916.2009965 · Cited by 117
A unified and discriminative model for query refinement
Guo, J., Xu, G., Li, H., & Cheng, X. (2008) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/1390334.1390400 · Cited by 115
Hassan, A., Shi, X., Craswell, N., & Ramsey, B. (2013) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/2505515.2505682 · Cited by 113
Characterizing and predicting search engine switching behavior
White, R. W., & Dumais, S. (2009) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/1645953.1645967 · Cited by 97
Self-citationStruggling or exploring? disambiguating long search sessions
Hassan, A., White, R. W., Dumais, S., & Wang, Y. (2014) · ACM International Conference on Web Search and Data Mining (WSDM) · doi:10.1145/2556195.2556221 · Cited by 94
Self-citationUnderstanding and Predicting Graded Search Satisfaction
Jiang, J., Awadallah, A. H., Shi, X., & White, R. W. (2015) · ACM International Conference on Web Search and Data Mining (WSDM) · doi:10.1145/2684822.2685319 · Cited by 86
Self-citationToward whole-session relevance: exploring intrinsic diversity in web search
Raman, K., Bennett, P. N., & Collins‐Thompson, K. (2013) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/2484028.2484089 · Cited by 86
Searching biases in large interactive document retrieval systems
Blair, D. C. (1980) · Journal of the American Society for Information Science · doi:10.1002/asi.4630310407 · Cited by 62
When web search fails, searchers become askers: understanding the transition
Liu, Q., Agichtein, E., Dror, G., Maarek, Y., & Szpektor, I. (2012) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/2348283.2348390 · Cited by 61
The re:search engine: simultaneous support for finding and re-finding
Teevan, J. B. (2007) · ACM Symposium on User Interface Software and Technology (UIST) · doi:10.1145/1294211.1294217 · Cited by 52
Leading people to longer queries
Agapie, E., Golovchinsky, G., & Qvarfordt, P. (2013) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/2470654.2481418 · Cited by 52
Evaluating the effectiveness of search task trails
Liao, Z., Song, Y., He, L.-W., & Huang, Y. (2012) · International World Wide Web Conference (WWW) · doi:10.1145/2187836.2187903 · Cited by 51
Task-aware query recommendation
Feild, H. A., & Allan, J. (2013) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/2484028.2484069 · Cited by 45
Measuring improvement in user search performance resulting from optimal search tips
Moraveji, N., Russell, D. M., Bien, J., & Mease, D. (2011) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/2009916.2009966 · Cited by 44
Modeling action-level satisfaction for search task satisfaction prediction
Wang, H., Song, Y., Chang, M.-W., He, X., Hassan, A., & White, R. W. (2014) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/2600428.2609607 · Cited by 42
Self-citationEvaluating web search using task completion time
Xu, Y., & Mease, D. (2009) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/1571941.1572073 · Cited by 36
The last click: why users give up information network navigation
Scaria, A. T., Philip, R. M., West, R., & Leskovec, J. (2014) · ACM International Conference on Web Search and Data Mining (WSDM) · doi:10.1145/2556195.2556232 · Cited by 36
Guo, Q., White, R. W., Zhang, Y., Anderson, B. J., & Dumais, S. (2011) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/2009916.2009964 · Cited by 35
Self-citationFighting search engine amnesia: reranking repeated results
Shokouhi, M., White, R. W., Bennett, P. N., & Radlinski, F. (2013) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/2484028.2484075 · Cited by 34
Self-citationIdentifying Web Search Query Reformulation using Concept based Matching
Hassan, A. (2013) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d13-1102 · Cited by 13
To hint or not: exploring the effectiveness of search hints for complex informational tasks
Savenkov, D., & Agichtein, E. (2014) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/2600428.2609523 · Cited by 10
Toward self-correcting search engines: using underperforming queries to improve search
Hassan, A., White, R. W., & Wang, Y. (2013) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/2484028.2484043 · Cited by 6
Self-citation
Cited by
Towards searching as a learning process: A review of current perspectives and future directions
Rieh, S. Y., Collins-Thompson, K., Hansen, P., & Lee, H.-J. (2016) · Journal of Information Science · doi:10.1177/0165551515615841 · Cited by 208
Online Evaluation for Information Retrieval
Hofmann, K., Li, L. C., & Radlinski, F. (2016) · now publishers, Inc. eBooks · doi:10.1561/1500000051 · Cited by 163
When does Relevance Mean Usefulness and User Satisfaction in Web Search?
Mao, J., Liu, Y., Zhou, K., Nie, J., Song, J., Zhang, M., Ma, S., Sun, J., & Luo, H. (2016) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/2911451.2911507 · Cited by 121
Sanchiz, M., Chevalier, A., & Amadieu, F. (2017) · Computers in Human Behavior · doi:10.1016/j.chb.2017.02.038 · Cited by 79
A Click Sequence Model for Web Search
Borisov, A., Wardenaar, M., Markov, I., & de Rijke, M. (2018) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3209978.3210004 · Cited by 71
Metadata
- Authors
- Odijk, D., White, R. W., Hassan Awadallah, A., & Dumais, S. T.
- Affiliations
- University of Amsterdam (NL)
- Microsoft (United States) (US)
- Microsoft Research, Redmond, WA, USA
- Venue
- Proceedings of the 24th ACM International on Conference on Information and Knowledge Management
- Publisher
- ACM
- Published
- 2015-10-17
- Pages
- 1551-1560
- ISSN
- 2155-0751 (ISSN-L)
- OpenAlex
- W2069916456
- Cited by
- 79
- References
- 48
- Citations by year
- 2026: 2 · 2025: 3 · 2024: 7 · 2023: 4 · 2022: 7 · 2021: 8 · 2020: 5 · 2019: 7 · 2018: 12 · 2017: 11 · 2016: 11 · 2015: 2
- FWCI
- 8.01
- Type
- conference paper
- Language
- en
- Open access
- green
- handle.uba.uva.nl
- Locations
- ACM International Conference on Information and Knowledge Management (CIKM) (publishedVersion)
- UvA-DARE (University of Amsterdam) (publishedVersion, other-oa)
- Data Archiving and Networked Services (DANS) (submittedVersion, other-oa)
- Funders
- Dutch national program COMMIT
- Topics
- Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
- Physical Sciences › Computer Science › Information Systems › Recommender Systems and Techniques
- Physical Sciences › Computer Science › Information Systems › Web Data Mining and Analysis
- Keywords
- web search, search success, search logs, search system design, user frustration, Bing search engine
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.