The DOI matches its Crossref record; 47 of the 48 references it cites were found in a registry or an index.
Struggling and Success in Web Search
Open accessAbstract
Web searchers sometimes struggle to find relevant information. Struggling leads to frustrating and dissatisfying search experiences, even if searchers ultimately meet their search objectives. Better understanding of search tasks where people struggle is important in improving search systems. We address this important issue using a mixed methods study using large-scale logs, crowd-sourced labeling, and predictive modeling. We analyze anonymized search logs from the Microsoft Bing Web search engine to characterize aspects of struggling searches and better explain the relationship between struggling and search success. To broaden our understanding of the struggling process beyond the behavioral signals in log data, we develop and utilize a crowd-sourced labeling methodology. We collect third-party judgments about why searchers appear to struggle and, if appropriate, where in the search task it became clear to the judges that searches would succeed (i.e., the pivotal query). We use our findings to propose ways in which systems can help searchers reduce struggling. Key components of such support are algorithms that accurately predict the nature of future actions and their anticipated impact on search outcomes. Our findings have implications for the design of search systems that help searchers struggle less and succeed more.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Citations
Cites
Classification and Regression Trees.
Gordon, A. D., Breiman, L., Friedman, J. H., Olshen, R. A., & Stone, C. J. (1984) · Biometrics · doi:10.2307/2530946 · Cited by 23,522
Classification and Regression Trees.
Ryzin, J. V., Breiman, L., Friedman, J. H., Olshen, R. A., & Stone, C. J. (1986) · Journal of the American Statistical Association · doi:10.2307/2288003 · Cited by 20,825
Classification And Regression Trees
Breiman, L., Friedman, J. H., Olshen, R. A., & Stone, C. J. (2017) · doi:10.1201/9781315139470 · Cited by 11,417
Satisfaction: a behavioral perspective on the consumer
(2010) · Choice Reviews Online · doi:10.5860/choice.47-6963 · Cited by 6,042
The Oxford Handbook of Positive Psychology
(2009) · doi:10.1093/oxfordhb/9780195187243.001.0001 · Cited by 2,696 · cited 1×
“Satisfaction has been studied extensively in a number of areas such as psychology [30] and commerce [33].”
BackgroundInter-Coder Agreement for Computational Linguistics
Artstein, R., & Poesio, M. (2008) · Computational Linguistics · doi:10.1162/coli.07-034-r2 · Cited by 1,665
Exploratory search: from finding to understanding
Marchionini, G. (2006) · Communications of the ACM · doi:10.1145/1121949.1121979 · Cited by 1,506
Methods for Evaluating Interactive Information Retrieval Systems with Users
Kelly, D. (2009) · Foundations and Trends® in Information Retrieval · doi:10.1561/1500000012 · Cited by 633
Generating query substitutions
Jones, R., Rey, B., Madani, O., & Greiner, W. (2006) · Proceedings of the 15th international conference on World Wide Web · doi:10.1145/1135777.1135835 · Cited by 618
Evaluating implicit measures to improve web search
Fox, S., Karnawat, K., Mydland, M., Dumais, S., & White, T. (2005) · ACM Transactions on Information Systems · doi:10.1145/1059981.1059982 · Cited by 412
Self-citation
Cited by
Towards searching as a learning process: A review of current perspectives and future directions
Rieh, S. Y., Collins-Thompson, K., Hansen, P., & Lee, H.-J. (2016) · Journal of Information Science · doi:10.1177/0165551515615841 · Cited by 208
Online Evaluation for Information Retrieval
Hofmann, K., Li, L. C., & Radlinski, F. (2016) · now publishers, Inc. eBooks · doi:10.1561/1500000051 · Cited by 163
When does Relevance Mean Usefulness and User Satisfaction in Web Search?
Mao, J., Liu, Y., Zhou, K., Nie, J., Song, J., Zhang, M., Ma, S., Sun, J., & Luo, H. (2016) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/2911451.2911507 · Cited by 121
Sanchiz, M., Chevalier, A., & Amadieu, F. (2017) · Computers in Human Behavior · doi:10.1016/j.chb.2017.02.038 · Cited by 79
A Click Sequence Model for Web Search
Borisov, A., Wardenaar, M., Markov, I., & de Rijke, M. (2018) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3209978.3210004 · Cited by 71
Task Supportive and Personalized Human-Large Language Model Interaction: A User Study
Wang, B., Liu, J., Karimnazarov, J., & Thompson, N. (2024) · Conference on Human Information Interaction and Retrieval · doi:10.1145/3627508.3638344 · Cited by 66
How Does Domain Expertise Affect Users’ Search Interaction and Outcome in Exploratory Search?
Mao, J., Liu, Y., Kando, N., Zhang, M., & Ma, S. (2018) · ACM Transactions on Information Systems · doi:10.1145/3223045 · Cited by 61
Interactive Intent Modeling for Exploratory Search
Ruotsalo, T., Peltonen, J., Eugster, M. J. A., Głowacka, D., Floréen, P., Myllymäki, P., Jacucci, G., & Kaski, S. (2018) · ACM Transactions on Information Systems · doi:10.1145/3231593 · Cited by 57
Just Give Me What I Want: How People Use and Evaluate Music Search
Hosey, C., Vujović, L., Thomas, B. S., Garcia-Gathright, J. I., & Thom, J. (2019) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300529 · Cited by 54
Query Reformulation in E-Commerce Search
Hirsch, S., Guy, I., Nus, A., Dagan, A., & Kurland, O. (2020) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3397271.3401065 · Cited by 52
Liu, M., Liu, Y., Mao, J., Luo, C., Zhang, M., & Ma, S. (2018) · International World Wide Web Conference (WWW) · doi:10.1145/3178876.3186065 · Cited by 51
A Context-aware Time Model for Web Search
Borisov, A., Markov, I., de Rijke, M., & Serdyukov, P. (2016) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/2911451.2911504 · Cited by 48
Liu, B., Wu, Y., Zhang, F., Liu, Y., Wang, Z., Li, C., Zhang, M., & Ma, S. (2022) · Information Processing & Management · doi:10.1016/j.ipm.2022.103051 · Cited by 41
Effects of LLM-based Search on Decision Making: Speed, Accuracy, and Overreliance
Spatharioti, S. E., Rothschild, D., Goldstein, D. G., & Hofman, J. M. (2025) · Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3714082 · Cited by 36
Did We Get It Right? Predicting Query Performance in E-commerce SearchPreprint
Kumar, R., Kumar, M., Shah, N., & Faloutsos, C. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1808.00239 · Cited by 36
Conversational vs Traditional: Comparing Search Behavior and Outcome in Legal Case Retrieval
Liu, B., Wu, Y., Liu, Y., Zhang, F., Shao, Y., Li, C., Zhang, M., & Ma, S. (2021) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3404835.3463064 · Cited by 35
Modeling the usefulness of search results as measured by information use
Vakkari, P., Völske, M., Potthast, M., Hagen, M., & Stein, B. (2019) · Information Processing & Management · doi:10.1016/j.ipm.2019.02.001 · Cited by 33
Sanchiz, M., Amadieu, F., Paubel, P., & Chevalier, A. (2019) · Behaviour and Information Technology · doi:10.1080/0144929x.2019.1642384 · Cited by 30
Liu, C., Zhang, X., & Huang, W. K. (2016) · Proceedings of the Association for Information Science and Technology · doi:10.1002/pra2.2016.14505301063 · Cited by 27
User Behaviour and Task Characteristics: A Field Study of Daily Information Behaviour
He, J., & Yılmaz, E. (2017) · Conference on Human Information Interaction and Retrieval · doi:10.1145/3020165.3020188 · Cited by 23
Understanding user behavior in job and talent search: an initial investigation
Spina, D., Maistro, M., Ren, Y., Sadeghi, S., Wong, W., Baldwin, T., Cavedon, L., Moffat, A., Sanderson, M., Scholer, F., et al. (2017) · RMIT Research Repository (RMIT University Library) · Cited by 19
Williams, K. B., & Zitouni, I. (2017) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3132847.3133035 · Cited by 19
Wang, B., & Liu, J. (2024) · ACM SIGIR International Conference on Theory of Information Retrieval (ICTIR) · doi:10.1145/3664190.3672520 · Cited by 18
Characterising Student Engagement Modes through Low-Level Activity Patterns
Gledson, A., Apaolaza, A., Barthold, S., Günther, F., Yu, H., & Vigo, M. (2021) · ACM Conference on User Modeling, Adaptation and Personalization (UMAP) · doi:10.1145/3450613.3456818 · Cited by 16
Yamamoto, Y. (2024) · Frontiers in Psychology · doi:10.3389/fpsyg.2024.1382234 · Cited by 15
Session search modeling by partially observable Markov decision process
Yang, G. H., Dong, X., Luo, J., & Zhang, S. (2017) · Information Retrieval · doi:10.1007/s10791-017-9316-8 · Cited by 15
Intent Mismatch Causes LLMs to Get Lost in Multi-Turn ConversationPreprint
Liu, G., Zhu, F., Feng, R., Ma, C., Wang, S., & Meng, G. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2602.07338 · Cited by 14
Uncovering Task Based Behavioral Heterogeneities in Online Search Behavior
Mehrotra, R., Bhattacharya, P., & Yılmaz, E. (2016) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/2911451.2914755 · Cited by 13
Automatic assessment of interactive OLAP explorations
Djedaini, M., Drushku, K., Labroche, N., Marcel, P., Peralta, V., & Verdeaux, W. (2018) · Information Systems · doi:10.1016/j.is.2018.06.008 · Cited by 13
Task Intelligence for Search and Recommendation
Shah, C. S., & White, R. W. (2021) · Synthesis lectures on information concepts, retrieval, and services · doi:10.2200/s01103ed1v01y202105icr074 · Cited by 11
Self-citationCoST: An annotated Data Collection for Complex Search
Dosso, C., Moreno, J. G., Chevalier, A., & Tamine, L. (2021) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3459637.3481998 · Cited by 10
Search task success evaluation by exploiting multi-view active semi-supervised learning
Chen, L., Fan, A., Shi, H., & Chen, G. (2019) · Information Processing & Management · doi:10.1016/j.ipm.2019.102180 · Cited by 10
Detecting User Focus in OLAP Analyses
Djedaini, M., Labroche, N., Marcel, P., & Peralta, V. (2017) · Lecture notes in computer science · doi:10.1007/978-3-319-66917-5_8 · Cited by 9
La visualidad algorítmica: una aproximación social a la visión artificial en la era post internet
Medina, F. R. C., & Marín, A. (2022) · Arte individuo y sociedad · doi:10.5209/aris.74664 · Cited by 9
What Can Task Teach Us About Query Reformulations?
Tamine, L., Lovón-Melgarejo, J., & Pinel-Sauvagnat, K. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-45439-5_42 · Cited by 9
A personalized ranking method based on inverse reinforcement learning in search engines
Karamiyan, F., Mahootchi, M., & Mohebi, A. (2024) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2024.108915 · Cited by 8
Automatic Semantic Network Generation from Unstructured Documents – The Options
Wanjawa, B., & Muchemi, L. (2018) · International Conference on Soft Computing & Machine Intelligence (ISCMI) · doi:10.1109/iscmi.2018.8703225 · Cited by 8
Competitive Advertising on Brand Search: Traffic Stealing, Adverse Selection and Customer ConfusionPreprint
Simonov, A., & Hill, S. (2018) · SSRN Electronic Journal · doi:10.2139/ssrn.3204394 · Cited by 8
A Proactive System for Supporting Users in Interactions with Large Language Models
Wang, B. (2024) · Conference on Human Information Interaction and Retrieval · doi:10.1145/3627508.3638325 · Cited by 7
Task Intelligence for Search and Recommendation
Shah, C. S., & White, R. W. (2021) · Synthesis lectures on information concepts, retrieval, and services · doi:10.1007/978-3-031-02326-2 · Cited by 7
Self-citationQuery Suggestion for Struggling Search by Struggling Flow Graph
Zebang, C., Yamamoto, T., & Tanaka, K. (2016) · IEEE/WIC/ACM International Conference on Web Intelligence · doi:10.1109/wi.2016.0040 · Cited by 6
A Click Sequence Model for Web SearchPreprint
Borisov, A., Wardenaar, M., Markov, I., & de Rijke, M. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1805.03411 · Cited by 6
Tannenbaum, B. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2607.18904 · Cited by 6
A Crowd-Powered Task Generation Method for Study of Struggling Search
Xu, L., & Zhou, X. (2021) · Data Science and Engineering · doi:10.1007/s41019-021-00171-3 · Cited by 5
Query Log Mining for Inferring User Tasks and Needs
Mehrotra, R., & Yılmaz, E. (2016) · Lecture notes in computer science · doi:10.1007/978-3-319-46131-1_36 · Cited by 5
A multi-view semi-supervised approach for task-level web search success evaluation
Fan, A., Chen, L., & Chen, G. (2017) · Information Sciences · doi:10.1016/j.ins.2017.12.003 · Cited by 5
Measurement of Users’ Experience on Online Platforms from Their Behavior Logs
Jain, D., Sinha, A. R., Gupta, D., Sheoran, N., & Khosla, S. (2018) · Lecture notes in computer science · doi:10.1007/978-3-319-93034-3_38 · Cited by 5
Generating Tasks for Study of Struggling Search
Xu, L., & Zhou, X. (2019) · Conference on Human Information Interaction and Retrieval · doi:10.1145/3295750.3298949 · Cited by 5
R-classify: Extracting research papers’ relevant concepts from a controlled vocabulary
Aggarwal, T., Salatino, A. A., Osborne, F., & Motta, E. (2022) · Software Impacts · doi:10.1016/j.simpa.2022.100444 · Cited by 5
Exploring the Value of Multi-View Learning for Session-Aware Query Representation
Ortiz, D., Moreno, J. G., Hubert, G., Pinel-Sauvagnat, K., & Tamine, L. (2022) · Findings of the Association for Computational Linguistics: NAACL · doi:10.18653/v1/2022.findings-naacl.23 · Cited by 4
How does Domain Expertise Affect Users' Search Processes in Exploratory Searches?
Mao, J., Liu, Y., Zhang, M., & Ma, S. (2016) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · Cited by 4
Query Expansion and Query Fuzzy with Large-Scale Click-through Data for Microblog Retrieval
Pang, W., & Du, J. (2019) · International Journal of Machine Learning and Computing · doi:10.18178/ijmlc.2019.9.3.799 · Cited by 4
Characterizing and Early Predicting User Performance for Adaptive Search Path Recommendation
Wang, B., & Liu, J. (2023) · Proceedings of the Association for Information Science and Technology · doi:10.1002/pra2.799 · Cited by 4
Ahmadvand, A., Arabzadeh, N., Kiseleva, J., Sanz, P. F., Deng, X., Jauhar, S. K., Gamon, M., Agichtein, E., Friend, N., & Aniruddha (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2205.00584 · Cited by 4
Surveys without Questions: A Reinforcement Learning Approach
Sinha, A. R., Jain, D., Sheoran, N., Khosla, S., & Sasidharan, R. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.3301257 · Cited by 3
Metric-agnostic Ranking Optimization
Ai, Q., Wang, X., & Bendersky, M. (2023) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3539618.3591935 · Cited by 3
Towards Understanding URL Resources in Recent Sina Weibo
Wan, Y., Li, P., Li, R., Zhou, M., Ye, Y., & Wang, B. (2016) · Lecture notes in computer science · doi:10.1007/978-3-319-48740-3_4 · Cited by 3
Chatbot to Facilitate Opinion Formation in Web Search
Okuse, Y., & Yamamoto, Y. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-35132-7_43 · Cited by 3
A user-centred approach to information retrieval
Aloteibi, S. (2020) · Apollo (University of Cambridge) · doi:10.17863/cam.77137 · Cited by 3
Right answers to wrong questions: The dysfunctional nature of information needs
Kilian, M. A., Elsweiler, D., & Ruthven, I. G. (2025) · Journal of the Association for Information Science and Technology · doi:10.1002/asi.70010 · Cited by 2
Extracting Searching as Learning Tasks Based on IBRT Approach
Li, P., Zhang, B., & Zhang, Y. (2022) · Applied Sciences · doi:10.3390/app12125879 · Cited by 2
Jacucci, G. (2016) · Computer · doi:10.1109/mc.2016.298 · Cited by 2
Modelling User Behaviour based on Process
Zhuang, M. (2017) · ACM Conference on User Modeling, Adaptation and Personalization (UMAP) · doi:10.1145/3079628.3079705 · Cited by 2
Investigating Per-user Time Sensitivity of Search Topics.
Nagar, J., & Suleman, H. (2017) · Cited by 2
TaskGenie: Crowd-Powered Task Generation for Struggling Search
Xu, L., Zhou, X., & Gadiraju, U. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-62008-0_1 · Cited by 2
Bota, H. (2018) · Enlighten: Theses (The University of Glasgow) · doi:10.5525/gla.thesis.38925 · Cited by 2
Improving searcher struggle detection via the reversal theory
Luo, J., Yang, Y., Nayak, V., & Yang, G. H. (2024) · Discover Computing · doi:10.1007/s10791-024-09492-z · Cited by 1
Analysis of Item Selection Behavior in Online Shopping
Nozaki, Y., Fumiya, W., & Satoh, T. (2018) · International Conference on Information Integration and Web-based Applications & Services (iiWAS) · doi:10.1145/3282373.3282849 · Cited by 1
Feature Modulation to Improve Struggle Detection in Web Search: A Psychological ApproachPreprint
Luo, J., Yang, Y., Nayak, V., & Yang, G. H. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2112.04711 · Cited by 1
Sanchiz, M. (2017) · theses.fr (ABES) · doi:10.70675/55c26d8fz5a4fz44cczbec8z3aa31a59793e · Cited by 1
What makes book search in social media complex
Koolen, M., Bogers, T., van den Bosch, A. P. J., & Kamps, J. (2015) · Radboud Repository (Radboud University)
Struggling and Success in Web Search: abstract
Odijk, D., White, R. W., Awadallah, A. H., & Dumais, S. (2015) · UvA-DARE (University of Amsterdam)
Self-citationWykorzystanie inwentyki w kształceniu umiejętności wyszukiwania informacji
Korycińska, P. (2020) · Jagiellonian University Repository (Jagiellonian University)
History-Aware Expansion and Fuzzy for Query Reformulation
Pang, W., & Duan, R. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-93049-3_19
Searcher Struggle Detection via the Reversal TheoryPreprint
Luo, J., Yang, Y. R., Nayak, V., & Yang, G. H. (2022) · Research Square · doi:10.21203/rs.3.rs-1762506/v1
Ahmed, A., Jain, G., Sharma, A., Hashim, M., & Raj, A. (2023) · Lecture notes in mechanical engineering · doi:10.1007/978-981-99-3033-3_17
User Interaction-Driven Refinement of Dense Retrieval Models
Goli, R. (2025) · Conference on Human Information Interaction and Retrieval · doi:10.1145/3698204.3716484
Korycińska, P. (2024) · Jagiellonian University Repository (Jagiellonian University) · doi:10.26106/jxnv-xy69
Automatic assessment of OLAP exploration quality
Djedaini, M. (2017) · theses.fr (ABES) · doi:10.70675/e3406782z5540z43b4zb4dfzd1fa211837c3
Metadata
- Authors
- Odijk, D., White, R. W., Hassan Awadallah, A., & Dumais, S. T.
- Affiliations
- University of Amsterdam (NL)
- Microsoft (United States) (US)
- Microsoft Research, Redmond, WA, USA
- Venue
- Proceedings of the 24th ACM International on Conference on Information and Knowledge Management
- Publisher
- ACM
- Published
- 2015-10-17
- Pages
- 1551-1560
- ISSN
- 2155-0751 (ISSN-L)
- OpenAlex
- W2069916456
- Cited by
- 79
- References
- 48
- Citations by year
- 2026: 2 · 2025: 3 · 2024: 7 · 2023: 4 · 2022: 7 · 2021: 8 · 2020: 5 · 2019: 7 · 2018: 12 · 2017: 11 · 2016: 11 · 2015: 2
- FWCI
- 8.01
- Type
- conference paper
- Language
- en
- Open access
- green
- handle.uba.uva.nl
- Locations
- ACM International Conference on Information and Knowledge Management (CIKM) (publishedVersion)
- UvA-DARE (University of Amsterdam) (publishedVersion, other-oa)
- Data Archiving and Networked Services (DANS) (submittedVersion, other-oa)
- Funders
- Dutch national program COMMIT
- Topics
- Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
- Physical Sciences › Computer Science › Information Systems › Recommender Systems and Techniques
- Physical Sciences › Computer Science › Information Systems › Web Data Mining and Analysis
- Keywords
- web search, search success, search logs, search system design, user frustration, Bing search engine
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.