The DOI matches its Crossref record; 15 of the 15 references it cites were found in a registry or an index.
State-Aware Meta-Evaluation of Evaluation Metrics in Interactive Information Retrieval
Abstract
In interactive IR (IIR), users often seek to achieve different goals (e.g. exploring a new topic, finding a specific known item) at different search iterations and thus may evaluate system performances differently. Without state-aware approach, it would be extremely difficult to simulate and achieve real-time adaptive search evaluation and recommendation. To address this gap, our work identifies users' task states from interactive search sessions and meta-evaluates a series of online and offline evaluation metrics under varying states based on a user study dataset consisting of 1548 unique query segments from 450 search sessions. Our results indicate that: 1) users' individual task states can be identified and predicted from search behaviors and implicit feedback; 2) the effectiveness of mainstream evaluation measures (measured based upon their respective correlations with user satisfaction) vary significantly across task states. This study demonstrates the implicit heterogeneity in user-oriented IR evaluation and connects studies on complex search tasks with evaluation techniques. It also informs future research on the design of state-specific, adaptive user models and evaluation metrics.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Citations
Cites
Variable selection using Mean Decrease Accuracy and Mean Decrease Gini based on Random Forest
Han, H., Guo, X., & Hua, Y. (2016) · IEEE International Conference on Software Engineering and Service Science (ICSESS) · doi:10.1109/icsess.2016.7883053 · Cited by 448
SIGIR Initiative to Implement ACM Artifact Review and Badging
Ferro, N., & Kelly, D. (2018) · ACM SIGIR Forum · doi:10.1145/3274784.3274786 · Cited by 69
Liu, J. (2021) · Information Processing & Management · doi:10.1016/j.ipm.2021.102522 · Cited by 63
Self-citationTask, Information Seeking Intentions, and User Behavior
Liu, J., Mitsui, M. R., Belkin, N. J., & Shah, C. S. (2019) · Conference on Human Information Interaction and Retrieval · doi:10.1145/3295750.3298922 · Cited by 59
Self-citationMeta-evaluation of Online and Offline Web Search Evaluation Metrics
Chen, Y., Zhou, K., Liu, Y., Zhang, M., & Ma, S. (2017) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3077136.3080804 · Cited by 57
Identifying and Predicting the States of Complex Search Tasks
Liu, J., Sarkar, S., & Shah, C. S. (2020) · Conference on Human Information Interaction and Retrieval · doi:10.1145/3343413.3377976 · Cited by 50
Self-citationInvestigating Cognitive Effects in Session-level Search User Satisfaction
Liu, M., Mao, J., Liu, Y., Zhang, M., & Ma, S. (2019) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3292500.3330981 · Cited by 46
Predicting Information Seeking Intentions from Search Behaviors
Mitsui, M. R., Liu, J., Belkin, N. J., & Shah, C. S. (2017) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3077136.3080737 · Cited by 42
Self-citationPeople, Interacting with Information1
Belkin, N. J. (2016) · ACM SIGIR Forum · doi:10.1145/2888422.2888424 · Cited by 36
Interest Development, Knowledge Learning, and Interactive IR
Liu, J., & Jung, Y. J. (2021) · Conference on Human Information Interaction and Retrieval · doi:10.1145/3406522.3446015 · Cited by 22
Self-citationAugmenting web search surrogates with images
Capra, R., Arguello, J., & Scholer, F. (2013) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/2505515.2505714 · Cited by 20
Adaptive Effort for Search Evaluation Metrics
Jiang, J., & Allan, J. (2016) · Lecture notes in computer science · doi:10.1007/978-3-319-30671-1_14 · Cited by 13
Data Requirements for Evaluation of Personalization of Information Retrieval - A Position PaperPreprint
Belkin, N. J., Hienert, D., Mayr, P., & Shah, C. S. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1809.02412 · Cited by 7
Jones, G. J. F., Belkin, N. J., Lawless, S., & Pasi, G. (2018) · ACM SIGIR Forum · doi:10.1145/3274784.3274796 · Cited by 5
A Meta-Evaluation of Evaluation Methods for Diversified Search
Kingrani, S. K., Mark Levene, A., & Zhang, D. (2018) · Lecture notes in computer science · doi:10.1007/978-3-319-76941-7_43 · Cited by 2
Flagged first, then by citations
Research works printed but not linked: 1
- Michael Cole , Jingjing Liu , Nicholas Belkin , Ralf Bierig , Jacek Gwizdka , C Liu , Jin Zhang , and X Zhang . 2009 . Usefulness as the criterion for evaluation of interactive information retrieval . Proc. HCIR (2009), 1--4. Michael Cole, Jingjing Liu, Nicholas Belkin, Ralf Bierig, Jacek Gwizdka, C Liu, Jin Zhang, and X Zhang. 2009. Usefulness as the criterion for evaluation of interactive information retrieval. Proc. HCIR (2009), 1--4.
Cited by
Sorted by citations
Toward Cranfield-inspired reusability assessment in interactive information retrieval evaluation
Liu, J. (2022) · Information Processing & Management · doi:10.1016/j.ipm.2022.103007 · Cited by 23
Self-citationA Reference-Dependent Model for Web Search Evaluation
Chen, N., Liu, J., & Sakai, T. (2023) · International World Wide Web Conference (WWW) · doi:10.1145/3543507.3583551 · Cited by 17
Self-citationLiu, J., & Shah, C. S. (2022) · ACM/IEEE Joint Conference on Digital Libraries · doi:10.1145/3529372.3530926 · Cited by 14
Self-citationConstructing and meta-evaluating state-aware evaluation metrics for interactive search systems
Markwald, M., Liu, J., & Yu, R. (2023) · Information Retrieval · doi:10.1007/s10791-023-09426-1 · Cited by 8
Self-citationSIGIR 2024 Workshop on Simulations for Information Access (Sim4IA 2024)
Schaer, P., Kreutz, C. K., Balog, K., Breuer, T., & Fuhr, N. (2024) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3626772.3657991 · Cited by 5
Matching Search Result Diversity with User Diversity Acceptance in Web Search Sessions
Liu, J., & Han, F. (2022) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3477495.3531880 · Cited by 5
Self-citationLiu, J., & Azzopardi, L. A. (2024) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3626772.3661382 · Cited by 5
Self-citationCharacterizing and Early Predicting User Performance for Adaptive Search Path Recommendation
Wang, B., & Liu, J. (2023) · Proceedings of the Association for Information Science and Technology · doi:10.1002/pra2.799 · Cited by 4
Self-citationChen, N., Fang, H., Liu, J., Wei, W., Sakai, T., & Wu, X.-M. (2026) · ACM International Conference on Web Search and Data Mining (WSDM) · doi:10.1145/3773966.3779384 · Cited by 3
Self-citationJung, Y. J., & Liu, J. (2024) · Aslib Journal of Information Management · doi:10.1108/ajim-01-2024-0041 · Cited by 3
Self-citationOnline and Offline Evaluation in Search Clarification
Tavakoli, L., Trippas, J. R., Zamani, H., Scholer, F., & Sanderson, M. (2024) · ACM Transactions on Information Systems · doi:10.1145/3681786 · Cited by 1
Towards Better Evaluating Multi-Query Sessions: A Measure Based on the Theory of Planned Behavior
Zhang, W., Zhang, F., Chen, J., & Lu, W. P. (2025) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3726302.3730096 · Cited by 1
Implications and New Directions for IR Research and Practices
Liu, J. (2023) · The information retrieval series · doi:10.1007/978-3-031-23229-9_7 · Cited by 1
Self-citationEvaluating Cognitive Biases in Conversational and Generative IIR: A Tutorial
Azzopardi, L. A., & Liu, J. (2024) · Annual International ACM SIGIR Conference on Research and Development in Information Retrieval in the Asia Pacific Region (SIGIR-AP) · doi:10.1145/3673791.3698437 · Cited by 1
Self-citationWang, B., & Liu, J. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2512.11096 · Cited by 1
Self-citationFormally Modeling Users in Information Retrieval
Liu, J. (2023) · The information retrieval series · doi:10.1007/978-3-031-23229-9_2
Self-citationFrom Rational Agent to Human with Bounded Rationality
Liu, J. (2023) · The information retrieval series · doi:10.1007/978-3-031-23229-9_3
Self-citationA Synthetic Search Session Generator for Task-Aware Information Seeking and Retrieval
Sarkar, S., & Shah, C. S. (2023) · ACM International Conference on Web Search and Data Mining (WSDM) · doi:10.1145/3539597.3573041
Chen, N., & Sakai, T. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2307.02936
Zhao, Y., Chen, Z., Zhang, F., Wu, J., Chen, Y., Xie, Q., & Yu, S. (2025) · Lecture notes in business information processing · doi:10.1007/978-3-031-94184-9_10
Metadata
- Authors
- Liu, J., & Yu, R.
- Affiliations
- University of Oklahoma (US)
- University of Bonn (DE)
- ORCID
- Venue
- Proceedings of the 30th ACM International Conference on Information & Knowledge Management
- Publisher
- ACM
- Published
- 2021-10-26
- Pages
- 3258-3262
- ISSN
- 2155-0751 (ISSN-L)
- OpenAlex
- W3208649236
- Semantic Scholar
- d5dce4f5f3b1898d4da3304778cf0231338d0233
- Cited by
- 20
- References
- 15
- Citations by year
- 2026: 1 · 2025: 3 · 2024: 5 · 2023: 8 · 2022: 3
- FWCI
- 2.69
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- ACM International Conference on Information and Knowledge Management (CIKM) (publishedVersion)
- Funders
- National Science Foundation (2106152)
- Topics
- Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
- Physical Sciences › Computer Science › Information Systems › Recommender Systems and Techniques
- Physical Sciences › Computer Science › Human-Computer Interaction › Usability and User Interface Design
- Keywords
- interactive information retrieval, meta-evaluation, evaluation metrics, user satisfaction, implicit feedback, search behavior, user-centered evaluation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
