CoChat
Sign Up
Cited by 52Information Retrieval and Search BehaviorRecommender Systems and TechniquesWeb Data Mining and Analysis

The DOI matches its Crossref record; 13 of the 13 references it cites were found in a registry or an index.

User Simulation for Evaluating Information Access Systems

Open access
Balog, Krisztian, Zhai, ChengXiang·2024·Foundations and Trends® in Information Retrieval
DOI 10.1561/1500000098

Published version of a preprint

Open this paper in CoChat — free accountOpen PDF

Abstract

Information access systems, such as search engines, recom-mender systems, and conversational assistants, have become integral to our daily lives as they help us satisfy our in-formation needs. However, evaluating the effectiveness of these systems presents a long-standing and complex scien-tific challenge. This challenge is rooted in the difficulty of assessing a system's overall effectiveness in assisting users to complete tasks through interactive support, and further exacerbated by the substantial variation in user behaviour and preferences. To address this challenge, user simulation emerges as a promising solution. This monograph focuses on providing a thorough under-standing of user simulation techniques designed specifically for evaluation purposes. We begin with a background of in-formation access system evaluation and explore the diverse applications of user simulation. Subsequently, we systemati-cally review the major research progress in user simulation, covering both general frameworks for designing user simu-lators, utilizing user simulation for evaluation, and specific models and algorithms for simulating user interactions with search engines, recommender systems, and conversational assistants. Realizing that user simulation is an interdisciplinary research topic, whenever possible, we attempt to establish connections with related fields, including machine learning, dialogue systems, user modeling, and economics. We end the monograph with a broad discussion of important future research directions, many of which extend beyond the evaluation of information access systems and are expected to have broader impact on how to evaluate interactive intelligent systems in general.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1561/1500000098

Cites

13 of 13 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • IR has spent decades building reproducible ways to study users and to evaluate systems that serve them, including simulation as a first-class methodology.

    “IR has spent decades building reproducible ways to study users and to evaluate systems that serve them, including simulation as a first-class methodology [4].”

    From Agentic Memory Management Is the Next Frontier of Information Retrieval (2026) · cites this paper as [4]

  • We can re-run the same model and task with a different memory state, enabling a more controlled counterfactual comparison than is usually possible after a person has experienced one condition.

    “We can re-run the same model and task with a different memory state, enabling a more controlled counterfactual comparison than is usually possible after a person has experienced one condition [4].”

    From Agentic Memory Management Is the Next Frontier of Information Retrieval (2026) · cites this paper as [4]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Flagged first, then by citations

Research works printed but not linked: 404

  1. Abbasiantaeb, 2024, Let the LLMs Talk: Simulating Human-to-Human Conversational QA via Zero-Shot LLM-to-LLM Interactions, Proceedings of the 17th ACM International Conference on Web Search and Data Mining. WSDM ‘24., 8, 10.1145/3616855.3635856 · DOI
  2. Abolghasemi, 2023, Retrievability Bias Estimation Using Synthetically Generated Queries, Proceedings of the 32nd ACM International Conference on Information and Knowledge Management. CIKM ‘23., 3712
  3. Afzali, 2023, User-SimCRS: A User Simulation Toolkit for Evaluating Conversational Recommender Systems, Proceedings of the Sixteenth ACM International Conference on Web Search and Data Mining, 1160, 10.1145/3539597.3573029 · DOI
  4. Aher, 2023, Using Large Language Models to Simulate Multiple Humans and Replicate Human Subject Studies, International Conference on Machine Learning. ICML ‘23., 337
  5. Aliannejadi, 2021, Analysing Mixed Initiatives and Search Strategies during Conversational Search, Proceedings of the 30th ACM International Conference on Information and Knowledge Management. CIKM ‘21., 16
  6. Aliannejadi, 2020, ConvAI3: Generating Clarifying Questions for Open-Domain Dialogue Systems (ClariQ), arXiv: 2009.11352 [cs.CL] · arXiv
  7. Aliannejadi, 2019, Asking Clarifying Questions in Open-Domain Information-Seeking Conversations, Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval. SIGIR ‘19., 475, 10.1145/3331184.3331265 · DOI
  8. Anand, 2020, Conversational Search (Dagstuhl Seminar 19461), Dagstuhl Reports., 9, 34
  9. Arguello, 2012, Task Complexity, Vertical Display and User Interaction in Aggregated Search, Proceedings of the 35th International ACM SIGIR Conference on Research and Development in Information Retrieval. SIGIR ‘12., 435, 10.1145/2348283.2348343 · DOI
  10. Arora, 2021, A Survey of Inverse Reinforcement Learning: Challenges, Methods and Progress, Artificial Intelligence., 297, 103, 10.1016/j.artint.2021.103500 · DOI
  11. Arvola, 2010, Expected Reading Effort in Focused Retrieval Evaluation, Information Retrieval., 13, 460, 10.1007/s10791-010-9133-9 · DOI
  12. Aula, 2005, Eye-Tracking Reveals the Personal Styles for Search Result Evaluation, Proceedings of the 2005 IFIP TC13 International Conference on Human-Computer Interaction. INTERACT ‘05., 1058
  13. Azzopardi, 2009, Query Side Evaluation: An Empirical Analysis of Effectiveness and Effort, Proceedings of the 32nd International ACM SIGIR Conference on Research and Development in Information Retrieval. SIGIR ‘09., 556, 10.1145/1571941.1572037 · DOI
  14. Azzopardi, 2011, The Economics in Interactive Information Retrieval, Proceedings of the 34th International ACM SIGIR Conference on Research and Development in Information Retrieval. SIGIR ‘11., 15, 10.1145/2009916.2009923 · DOI
  15. Azzopardi, 2014, Modelling Interaction with Economic Models of Search, Proceedings of the 37th International ACM SIGIR Conference on Research and Development in Information Retrieval. SIGIR ‘14., 3, 10.1145/2600428.2609574 · DOI
  16. Azzopardi, 2018, Con-ceptualizing Agent-Human Interactions During the Conversational Search Process, Proceedings of the 2nd International Workshop on Conversational Approaches to Information Retrieval. CAIR ‘18.
  17. Azzopardi, 2011, Report on the SIGIR 2010 Workshop on the Simulation of Interaction, SIGIR Forum., 44, 35, 10.1145/1924475.1924484 · DOI
  18. Azzopardi, 2021, ERR is Not C/W/L: Exploring the Relationship Between Expected Reciprocal Rank and Other Metrics, Proceedings of the 2021 ACM SIGIR International Conference on Theory of Information Retrieval. ICTIR ‘21., 231, 10.1145/3471158.3472239 · DOI
  19. Azzopardi, 2006, Automatic Construction of Known-Item Finding Test Beds, Proceedings of the 29th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. SIGIR ‘06., 603
  20. Azzopardi, 2007, Building Simulated Queries for Known-Item Topics: An Analysis Using Six European Languages, Proceedings of the 30th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. SIGIR ‘07., 455
  21. Azzopardi, 2018, Measuring the Utility of Search Engine Result Pages: An Information Foraging Based Measure, The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval. SIGIR ‘18., 605, 10.1145/3209978.3210027 · DOI
  22. Azzopardi, 2019, Cwl_eval: An Evaluation Tool for Information Retrieval, Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval. SIGIR ‘19., 1321, 10.1145/3331184.3331398 · DOI
  23. Azzopardi, 2020, Data-Driven Evaluation Metrics for Heterogeneous Search Engine Result Pages, Proceedings of the 2020 Conference on Human Information Interaction and Retrieval. CHIIR ‘20., 213, 10.1145/3343413.3377959 · DOI
  24. Azzopardi, 2015, An Analysis of Theories of Search and Search Behavior, Proceedings of the 2015 International Conference on The Theory of Information Retrieval. ICTIR ‘15., 81
  25. Azzopardi, 2016, An Analysis of the Cost and Benefit of Search Interactions, Proceedings of the 2016 ACM International Conference on the Theory of Information Retrieval. ICTIR ‘16., 59
  26. Balog, 2021, Conversational AI from an Information Retrieval Perspective: Remaining Challenges and a Case for User Simulation, Proceedings of the 2nd International Conference on Design of Experimental Search and Information REtrieval Systems. DESIRES ‘21., 80
  27. Balog, 2019, Personal Knowledge Graphs: A Research Agenda, Proceedings of the 2019 ACM SIGIR International Conference on Theory of Information Retrieval. ICTIR ‘19., 217, 10.1145/3341981.3344241 · DOI
  28. Balog, 2022, Report on the 1st Simulation for Information Retrieval Workshop (Sim4IR 2021) at SIGIR 2021, SIGIR Forum., 55, 1, 10.1145/3527546.3527559 · DOI
  29. Baskaya, 2014, Simulating Search Sessions in Interactive Information Retrieval Evaluation, PhD thesis. University of Tampere
  30. Baskaya, 2011, Simulating Simple and Fallible Relevance Feedback, Proceedings of the 33rd European Conference on IR Research. ECIR ‘11., 593
  31. Baskaya, 2012, Time Drives Interaction: Simulating Sessions in Diverse Searching Environments, Proceedings of the 35th International ACM SIGIR Conference on Research and Development in Information Retrieval. SIGIR ‘12., 105, 10.1145/2348283.2348301 · DOI
  32. Baskaya, 2013, Modeling Behavioral Factors in Interactive Information Retrieval, Proceedings of the 22nd ACM International Conference on Information and Knowledge Management. CIKM ‘13., 2297
  33. Bates, 1989, The Design of Browsing and Berrypicking Techniques for the Online Search Interface, Online Review., 13, 407, 10.1108/eb024320 · DOI
  34. Belkin, 1992, Information Filtering and Information Retrieval: Two Sides of the Same Coin?, Communications of the ACM., 35, 29, 10.1145/138859.138861 · DOI
  35. Belkin, 1982, ASK for In-formation Retrieval: Part I. Background and Theory, Journal of Documentation., 10.1108/eb026722 · DOI
  36. Bellogin, 2011, Precision-oriented Evaluation of Recommender Systems: An Algorithmic Comparison, Proceedings of the Fifth ACM Conference on Recommender Systems. RecSys ‘11., 333, 10.1145/2043932.2043996 · DOI
  37. Bendada, 2020, Carousel Personalization in Music Streaming Apps with Contextual Bandits, Proceedings of the 14th ACM Conference on Recommender Systems. RecSys ‘20., 420, 10.1145/3383313.3412217 · DOI
  38. Bernard, 2023, MG-ShopDial: A Multi-Goal Conversational Dataset for e-Commerce, Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval. SIGIR ‘23., 2775, 10.1145/3539618.3591883 · DOI
  39. Bernard, 2024, Identifying Breakdowns in Conversational Recommender Systems using User Simulation, Proceedings of the 6th International Conference on Conversational User Interfaces. CUI ‘24.
  40. Berryman, 2006, What Defines “Enough” Information? How Policy Workers Make Judgements and Decisions during Information Seeking: Preliminary Results from an Exploratory Study, Information Research., 11
  41. Bi, 2021, Asking Clarifying Questions Based on Negative Feedback in Conversational Search, Proceedings of the 2021 ACM SIGIR International Conference on Theory of Information Retrieval. ICTIR ‘21., 157, 10.1145/3471158.3472232 · DOI
  42. Bignold, 2021, An Evaluation Methodology for Interactive Reinforcement Learning with Simulated Users, Biomimetics., 6, 13, 10.3390/biomimetics6010013 · DOI
  43. Birchler, 2007, Information Economics.
  44. Blunt, 1965, An Information Retrieval System Model, Tech. rep. HRB-SINGER Inc.
  45. Bollen, 2012, Remembering the Stars? Effect of Time on Preference Retrieval from Memory, Proceedings of the Sixth ACM Conference on Recommender Systems. RecSys ‘12., 217, 10.1145/2365952.2365998 · DOI
  46. Borgman, 1996, Why are Online Catalogs Still Hard to Use?, Journal of the American Society for Information Science., 47, 493, 10.1002/(SICI)1097-4571(199607)47:7<493::AID-ASI3>3.0.CO;2-P · DOI
  47. Bota, 2016, Playing Your Cards Right: The Effect of Entity Cards on Search Behaviour and Workload, Proceedings of the 2016 ACM on Conference on Human Information Interaction and Retrieval. CHIIR ‘16., 131
  48. Boyce, 1982, Beyond Topicality: A Two Stage View of Relevance and the Retrieval Process, Information Processing & Management., 18, 105, 10.1016/0306-4573(82)90033-4 · DOI
  49. Brennan, 2014, The Effect of Cognitive Abilities on Information Search for Tasks of Varying Levels of Complexity, Proceedings of the 5th Information Interaction in Context Symposium. IIiX ‘14., 165, 10.1145/2637002.2637022 · DOI
  50. Breuer, 2022, Validating Simulations of User Query Variants, Proceedings of the 44th European Conference on IR Research. ECIR ‘22., 80
  51. Broder, 2002, A Taxonomy of Web Search, SIGIR Forum., 36, 3, 10.1145/792550.792552 · DOI · Paper page
  52. Brown, 2020, Language Models are Few-Shot Learners, Advances in Neural Information Processing Systems. NeurIPS ‘20., 1877
  53. Browne, 2005, Stopping Rule Use During Web-Based Search, Proceedings of the 38th Annual Hawaii International Conference on System Sciences., 9, 271b, 10.1109/HICSS.2005.556 · DOI
  54. Browne, 2007, Cognitive Stopping Rules for Terminating Information Search in Online Tasks, MIS Quarterly., 31, 89, 10.2307/25148782 · DOI
  55. Buckley, 1999, The TREC-8 Query Track, Proceedings of The Eighth Text REtrieval Conference. TREC ‘99., 10.6028/NIST.SP.500-246.query-overview · DOI
  56. Budzianowski, 2018, MultiWOZ - A Large-Scale Multi-Domain Wizard-of-Oz Dataset for Task-Oriented Dialogue Modelling, Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. EMNLP ‘18., 5016, 10.18653/v1/D18-1547 · DOI · Paper page
  57. Bunt, 2020, The ISO Standard for Dialogue Act Annotation, Second Edition, Proceedings of the Twelfth Language Resources and Evaluation Conference. LREC ‘20., 549
  58. Bunt, 2017, Multimodal Interaction with W3C Standards: Toward Natural User Interfaces to Everything., 109, 10.1007/978-3-319-42816-1_6 · DOI
  59. Burnell, 2023, Rethink Reporting of Evaluation Results in AI, Science., 380, 136, 10.1126/science.adf6369 · DOI · Paper page
  60. Camara, 2022, Searching, Learning, and Subtopic Ordering: A Simulation-Based Analysis, Proceedings of the 44th European Conference on IR Research. ECIR ‘22., 142
  61. Card, 2001, Information Scent as a Driver of Web Behavior Graphs: Results of a Protocol Analysis Method for Web Usability, Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. CHI ‘01., 498
  62. Carterette, 2011, System Effectiveness, User Models, and User Utility: A Conceptual Framework for Investigation, Proceedings of the 34th International ACM SIGIR Conference on Research and Development in Information Retrieval. SIGIR ‘11., 903, 10.1145/2009916.2010037 · DOI
  63. Carterette, 2015, Dynamic Test Collections for Retrieval Evaluation, Proceedings of the 2015 International Conference on The Theory of Information Retrieval. ICTIR ‘15., 91
  64. Carterette, 2016, Evaluating Retrieval over Sessions: The TREC Session Track 2011-2014, Proceedings of the 39th International ACM SIGIR Conference on Research and Development in Information Retrieval. SIGIR ‘16., 685, 10.1145/2911451.2914675 · DOI
  65. Carterette, 2011, Simulating Simple User Behavior for System Effectiveness Evaluation, Proceedings of the 20th ACM International Conference on Information and Knowledge Management. CIKM ‘11., 611
  66. Chaney, 2021, Recommendation System Simulations: A Discussion of Two Key Challenges, arXiv: 2109.02475 [cs.IR] · arXiv
  67. Chapelle, 2009, Expected Reciprocal Rank for Graded Relevance, Proceedings of the 18th ACM Conference on Information and Knowledge Management. CIKM ‘09., 621, 10.1145/1645953.1646033 · DOI
  68. Charnov, 1976, Optimal Foraging, the Marginal Value Theorem, Theoretical Population Biology., 9, 129, 10.1016/0040-5809(76)90040-X · DOI · Paper page
  69. Chen, 2012, Beyond Ten Blue Links: Enabling User Click Modeling in Federated Web Search, Proceedings of the Fifth ACM International Conference on Web Search and Data Mining. WSDM ‘12., 463, 10.1145/2124295.2124351 · DOI
  70. Chen, 2012, Critiquing-based Recommenders: Survey and Emerging Trends, User Modeling and User-Adapted Interaction., 22, 125, 10.1007/s11257-011-9108-6 · DOI · Paper page
  71. Chen, 2021, A Survey of Deep Reinforcement Learning in Recommender Systems: A Systematic Review and Future Directions, arXiv: 2109.03540 [cs.IR] · arXiv
  72. Cheng, 2016, Wide & Deep Learning for Recommender Systems, Proceedings of the 1st Workshop on Deep Learning for Recommender Systems., 7, 10.1145/2988450.2988454 · DOI
  73. Cheng, 2022, Is MultiWOZ a Solved Task? An Interactive TOD Evaluation Framework with User Simulator, Findings of the Association for Computational Linguistics: EMNLP 2022. EMNLP ‘22., 124
  74. Chi, 2003, The Bloodhound Project: Automating Discovery of Web Usability Issues using the InfoScent™ Simulator, Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. CHI ‘03., 505
  75. Chuklin, 2015, Click Models for Web Search., 10.1007/978-3-031-02294-4 · DOI · Paper page
  76. Chuklin, 2012, Good Abandonments in Factoid Queries, Proceedings of the 21st International Conference on World Wide Web. WWW ‘12 Companion., 483, 10.1145/2187980.2188088 · DOI
  77. Clarke, 2011, Overview of the TREC 2011 Web Track, Proceedings of The Twentieth Text REtrieval Conference. TREC ‘11., 10.6028/NIST.SP.500-296.web-overview · DOI
  78. Clarke, 2008, Novelty and Diversity in Information Retrieval Evaluation, Proceedings of the 31st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. SIGIR ‘08., 659
  79. Cleverdon, 1968, Factors Determining the Performance of Indexing Systems, Aslib Cranfield Research Project, Cranfield, England
  80. Coleman, 1975, A Computer Readability Formula Designed for Machine Scoring, Journal of Applied Psychology., 60, 283, 10.1037/h0076540 · DOI
  81. Cooper, 1973, A Simulation Model of an Information Retrieval System, Information Storage and Retrieval., 9, 13, 10.1016/0020-0271(73)90004-1 · DOI
  82. Cooper, 1968, Expected Search Length: A Single Measure of Retrieval Effectiveness based on the Weak Ordering Action of Retrieval Systems, American Documentation., 19, 30, 10.1002/asi.5090190108 · DOI
  83. Cooper, 1973, On Selecting a Measure of Retrieval Effectiveness Part II. Implementation of the Philosophy, Journal of the American Society for Information Science., 24, 413, 10.1002/asi.4630240603 · DOI
  84. Cosijn, 2000, Dimensions of Relevance, Information Processing & Management., 36, 533, 10.1016/S0306-4573(99)00072-2 · DOI
  85. Craswell, 2008, An Experimental Comparison of Click Position-Bias Models, Proceedings of the 2008 International Conference on Web Search and Data Mining. WSDM ‘08., 87
  86. Cremonesi, 2010, Performance of Recommender Algorithms on Top-N Recommendation Tasks, Proceedings of the Fourth ACM Conference on Recommender Systems. RecSys ‘10., 39, 10.1145/1864708.1864721 · DOI
  87. Culpepper, 2018, Research Frontiers in Information Retrieval: Report from the Third Strategic Workshop on Information Retrieval in Lorne (SWIRL 2018), SIGIR Forum., 52, 34, 10.1145/3274784.3274788 · DOI
  88. Dalton, 2020, CAsT-19: A Dataset for Conversational Information Seeking, Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval. SIGIR ‘20., 1985, 10.1145/3397271.3401206 · DOI
  89. Davidson, 2023, User Simulation with Large Language Models for Evaluating Task-Oriented Dialogue, arXiv: 2309.13233 [cs.CL] · arXiv
  90. De Mey, 1977, The Cognitive Viewpoint: Its Development and its Scope, Communication & Cognition., 10, 7
  91. Deldjoo, 2021, Towards Multi-Modal Conversational Information Seeking, Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval. SIGIR ‘21., 1577, 10.1145/3404835.3462806 · DOI
  92. Deriu, 2021, Survey on Evaluation Methods for Dialogue Systems, Artificial Intelligence Review., 54, 755, 10.1007/s10462-020-09866-x · DOI
  93. Devlin, 2019, BERT: Pre-training of Deep Bidirectional Transformers for Language Un-derstanding, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers). NAACL ‘19., 4171
  94. Diaz, 2009, Adaptation of Offline Vertical Selection Predictions in the Presence of User Feedback, Proceedings of the 32nd International ACM SIGIR Conference on Research and Development in Information Retrieval. SIGIR ‘09., 323, 10.1145/1571941.1571998 · DOI
  95. Diaz, 2013, Robust Models of Mouse Movement on Dynamic Web Search Results Pages, Proceedings of the 22nd ACM International Conference on Information and Knowledge Management. CIKM ‘13., 1451
  96. Diriye, 2012, Leaving so Soon? Understanding and Predicting Web Search Abandonment Ra-tionales, Proceedings of the 21st ACM International Conference on Information and Knowledge Management. CIKM ‘12., 1025
  97. Dodge, 2016, Evaluating Prerequisite Qualities for Learning End-to-End Dialog Systems, 4th International Conference on Learning Representations. ICLR ‘16.
  98. Dorga, 2015, Implementation and use of Simulated Students for Test and Validation of new Adaptive Educational Systems: a Practical Insight, International Journal of Artificial Intelligence in Education., 25, 319, 10.1007/s40593-015-0037-0 · DOI
  99. Dumais, 2003, Stuff I've Seen: A System for Personal Information Retrieval and Re-use, Proceedings of the 26th Annual International ACM SIGIR Conference on Research and Development in Informaion Retrieval. SIGIR ‘03., 72
  100. Dumais, 2010, Individual Dif-ferences in Gaze Patterns for Web Search, Proceedings of the Third Symposium on Information Interaction in Context. IliX ‘10., 185, 10.1145/1840784.1840812 · DOI
  101. Dupret, 2008, A User Browsing Model to Predict Search Engine Click Data from Past Observations, Proceedings of the 31st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. SIGIR ‘08., 331
  102. Dzyabura, 2013, Not by Search Alone: How Recommendations Complement Search Results, Proceedings of the 7th ACM Conference on Recommender Systems. RecSys ‘13., 371, 10.1145/2507157.2507231 · DOI
  103. Eckert, 1997, User Modeling for Spoken Dialogue System Evaluation, 1997 IEEE Workshop on Automatic Speech Recognition and Understanding Proceedings., 80, 10.1109/ASRU.1997.658991 · DOI
  104. Eickhoff, 2014, Lessons from the Journey: A Query Log Analysis of within-Session Learning, Proceedings of the 7th ACM International Conference on Web Search and Data Mining. WSDM ‘14., 223, 10.1145/2556195.2556217 · DOI
  105. Ekstrand, 2021, SimuRec: Workshop on Synthetic Data and Simulation Methods for Recommender Systems Research, Proceedings of the 15th ACM Conference on Recommender Systems. RecSys ‘21., 803
  106. El Asri, 2016, A Sequence-to-Sequence Model for User Simulation in Spoken Dialogue Systems, Proceedings of Interspeech 2016., 1151, 10.21437/Interspeech.2016-1175 · DOI
  107. Ellis, 1989, A Behavioural Approach to Information Retrieval System Design, Journal of Documentation., 45, 171, 10.1108/eb026843 · DOI · Paper page
  108. Faggioli, 2023, Perspectives on Large Language Models for Relevance Judgment, Proceedings of the 2023 ACM SIGIR International Conference on Theory of Information Retrieval. ICTIR ‘23., 39
  109. Felicioni, 2021, A Methodology for the Offline Evaluation of Recommender Systems in a User Interface with Multiple Carousels, Adjunct Proceedings of the 29th ACM Conference on User Modeling, Adaptation and Personalization. UMAP ‘21., 10, 10.1145/3450614.3461680 · DOI
  110. Ferrari Dacrema, 2022, Offline Evaluation of Recommender Systems in a User Interface With Multiple Carousels, Frontiers in Big Data., 5
  111. Ferraro, 2021, Break the Loop: Gender Imbalance in Music Recommenders, Proceedings of the 2021 Conference on Human Information Interaction and Retrieval. CHIIR ‘21., 249
  112. Ferro, 2019, Information Retrieval Evaluation in a Changing World Lessons Learned from 20 Years of CLEF., 10.1007/978-3-030-22948-1 · DOI
  113. Fleder, 2007, Recommender Systems and Their Impact on Sales Diversity, Proceedings of the 8th ACM Conference on Electronic Commerce. EC ‘07., 192, 10.1145/1250910.1250939 · DOI
  114. Fleder, 2009, Blockbuster Culture's Next Rise or Fall: The Impact of Recommender Systems on Sales Diversity, Management Science., 55, 697, 10.1287/mnsc.1080.0974 · DOI
  115. Friedman, 2023, Leveraging Large Language Models in Conversational Recommender Systems, arXiv: 2305.07961 [cs.IR] · arXiv
  116. Fu, 2007, SNIF-ACT: A Cognitive Model of User Navigation on the World Wide Web, Human-Computer Interaction., 22, 355, 10.1080/07370020701638806 · DOI
  117. Fuhr, 2008, A Probability Ranking Principle for Interactive Information Retrieval, Information Retrieval., 11, 251, 10.1007/s10791-008-9045-0 · DOI
  118. Gao, 2023, Large Language Models Empowered Agent-based Modeling and Simulation: A Survey and Perspectives, arXiv: 2312.11970 [cs.AI] · arXiv
  119. Gao, 2024, Simulating Human Society with Large Language Model Agents: City, Social Media, and Economic System, Companion Proceedings of the ACM on Web Conference 2024., 1290, 10.1145/3589335.3641253 · DOI
  120. Gao, 2021, Advances and Challenges in Conversational Recommender Systems: A Survey, AI Open., 2, 100, 10.1016/j.aiopen.2021.06.002 · DOI
  121. Gao, 2019, Neural Approaches to Con-versational AI, Foundations and Trends in Information Retrieval., 13, 127, 10.1561/1500000074 · DOI
  122. Gemmis, 2015, Semantics-Aware Content-Based Recommender Systems, Recommender Systems Handbook., 119, 10.1007/978-1-4899-7637-6_4 · DOI
  123. Gettys, 1979, Hypothesis Plausibility and Hypothesis Generation, Organizational Behavior and Human Performance., 24, 93, 10.1016/0030-5073(79)90018-7 · DOI
  124. Ghanem, 2022, Balancing Con-sumer and Business Value of Recommender Systems: A Simulation-based Analysis, Electronic Commerce Research and Applications., 55, 101, 10.1016/j.elerap.2022.101195 · DOI
  125. Ghosh, 2018, Searching as Learning: Exploring Search Behavior and Learning Outcomes in Learning-related Tasks, Proceedings of the 2018 Conference on Human Information Interaction & Retrieval. CHIIR ‘18., 22
  126. Gordon, 1990, Evaluating the Effectiveness of Information Retrieval Systems Using Simulated Queries, Journal of the American Society for Information Science., 41, 313, 10.1002/(SICI)1097-4571(199007)41:5<313::AID-ASI1>3.0.CO;2-G · DOI
  127. Griffiths, 1976, The Computer Simulation of Information Retrieval Systems, PhD thesis. University of London (University College)
  128. Griol, 2013, An Automatic Dialog Simulation Technique to Develop and Evaluate Interactive Conver-sational Agents, Applied Artificial Intelligence., 27, 759, 10.1080/08839514.2013.835230 · DOI
  129. Günther, 2021, Assessing Query Suggestions for Search Session Simulation, Joint Proceedings of the Causality in Search and Recommendation (CSR) and Simulation of Information Retrieval Evaluation (Sim4IR) Workshops 2021., 38
  130. Guo, 2009, Click Chain Model in Web Search, Proceedings of the 18th International Conference on World Wide Web. WWW ‘09., 11, 10.1145/1526709.1526712 · DOI
  131. Guo, 2017, DeepFM: A Factorization-Machine Based Neural Network for CTR Prediction, Proceedings of the 26th International Joint Conference on Artificial Intelligence. IJCAI'17., 1725
  132. Guo, 2011, Why Searchers Switch: Understanding and Predicting Engine Switching Rationales, Proceedings of the 34th international ACM SIGIR conference on Research and development in Information Retrieval., 335
  133. Guo, 2018, Dialog-based Interactive Image Retrieval, Advances in Neural Information Processing Systems. NeurIPS ‘18.
  134. Gür, 2018, User Modeling for Task Oriented Dialogues, 2018 IEEE Spoken Language Technology Workshop. SLT ‘18., 900, 10.1109/SLT.2018.8639652 · DOI · Paper page
  135. Harman, 1992, Relevance Feedback Revisited, Proceedings of the 15th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. SIGIR ‘92., 1
  136. Harman, 1995, Overview of the Second Text Retrieval Conference (TREC-2), Information Processing & Management., 31, 271, 10.1016/0306-4573(94)00047-7 · DOI
  137. Hauptmann, 2020, MuCAI'20: 1st International Workshop on Multimodal Conversational AI, Proceedings of the 28th ACM International Conference on Multimedia. MM ‘20., 4767
  138. Hausman, 2011, Preference, Value, Choice, and Welfare., 10.1017/CBO9781139058537 · DOI
  139. Hawking, 2020, Simulating Information Retrieval Test Collections., 10.1007/978-3-031-02323-1 · DOI
  140. Hayati, 2020, INSPIRED: Toward Sociable Recommendation Dialog Systems, Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing. EMNLP ‘20., 8142, 10.18653/v1/2020.emnlp-main.654 · DOI
  141. Hazrati, 2022, Recommender Systems Effect on the Evolution of Users' Choices Distribution, Information Processing & Management., 59, 10.1016/j.ipm.2021.102766 · DOI
  142. Hazrati, 2024, Choice Models and Recommender Systems Effects on users' Choices, User Modeling and User-Adapted Interaction., 34, 109, 10.1007/s11257-023-09366-x · DOI
  143. Hearst, 2009, Search User Interfaces., 10.1017/CBO9781139644082 · DOI · Paper page
  144. Hersh, 2000, Do Batch and User Evaluations Give the Same Results?, Proceedings of the 23rd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. SIGIR ‘00., 17, 10.1145/345508.345539 · DOI
  145. Hofmann, 2016, Online Evaluation for Information Retrieval, Foundations and Trends in Information Retrieval., 10, 1, 10.1561/1500000051 · DOI
  146. Hofmann, 2011, A Probabilistic Method for Inferring Preferences from Clicks, Proceedings of the 20th ACM International Conference on Information and Knowledge Management. CIKM ‘11., 249
  147. Hopfgartner, 2019, Information Retrieval Evaluation in a Changing World - Lessons Learned from 20 Yea.rs of CLEF., 511
  148. Hu, 2014, Convolutional Neural Network Architectures for Matching Natural Language Sentences, Proceedings of the 27th International Conference on Neural Information Processing Systems - Volume 2. NIPS ‘14., 2042
  149. Hu, 2023, Unlocking the Potential of User Feedback: Leveraging Large Language Model as User Simulators to Enhance Dialogue System, Proceedings of the 32nd ACM International Conference on Information and Knowledge Management. CIKM ‘23., 3953
  150. Huang, 2012, Improving Searcher Models Using Mouse Cursor Activity, Proceedings of the 35th International ACM SIGIR Conference on Research and Development in Information Retrieval. SIGIR ‘12., 195, 10.1145/2348283.2348313 · DOI
  151. Hussein, 2017, Imitation Learning: A Survey of Learning Methods, ACM Computing Surveys., 50, 1, 10.1145/3054912 · DOI · Paper page
  152. Ie, 2019, RecSim: A Configurable Simulation Platform for Recommender Systems, arXiv: 1909.04847 [cs.LG] · arXiv
  153. Ingwersen, 1982, Search Procedures in the Library—Analysed from the Cognitive Point of View, Journal of Documentation., 38, 165, 10.1108/eb026727 · DOI
  154. Ingwersen, 2005, The Turn: Integration of Information Seeking and Retrieval in Context, 18
  155. Jagerman, 2018, OpenSearch: Lessons Learned from an Online Evaluation Campaign, Journal of Data and Information Quality., 10, 13:1, 10.1145/3239575 · DOI
  156. Jameson, 2014, Choice Architecture for Human-Computer Interaction, Foundations and Trends in Human-Computer Interaction., 7, 1, 10.1561/1100000028 · DOI
  157. Jameson, 2022, Individual and Group Decision Making and Recommender Systems, Recommender Systems Handbook, 789, 10.1007/978-1-0716-2197-4_21 · DOI
  158. Jameson, 2015, Human Decision Making and Recommender Systems, Recommender Systems Handbook., 611, 10.1007/978-1-4899-7637-6_18 · DOI
  159. Jannach, 2015, What Recommenders Recommend: An Analysis of Recommendation Biases and Possible Countermeasures, User Modeling and User-Adapted Interaction., 25, 427, 10.1007/s11257-015-9165-3 · DOI
  160. Jannach, 2021, A Survey on Conversational Recommender Systems, ACM Computing Surveys., 54, 1, 10.1145/3453154 · DOI · Paper page
  161. Jannach, 2017, User Control in Recommender Systems: Overview and Interaction Challenges, Proceedings of the 17th International Conference on Electronic Commerce and Web Technologies. EC-Web ‘16., 21
  162. Jansen, 2007, Defining a Session on Web Search Engines: Research Articles, Journal of the American Society for Information Science and Technology., 58, 862, 10.1002/asi.20564 · DOI
  163. Jansen, 2000, Real Life, Real Users, and Real Needs: A Study and Analysis of User Queries on the Web, Information Processing & Management., 36, 207, 10.1016/S0306-4573(99)00056-4 · DOI
  164. Järvelin, 2008, Discounted Cumulated Gain Based Evaluation of Multiple-Query IR Sessions, Proceedings of the 30th European Conference on Advances in Information Retrieval. ECIR ‘08., 4
  165. Joachims, 2007, Evaluating the Accuracy of Implicit Feedback from Clicks and Query Reformulations in Web Search, ACM Transactions on Information Systems., 25, 10.1145/1229179.1229181 · DOI
  166. Jordan, 2006, Using Controlled Query Generation to Evaluate Blind Relevance Feedback Algorithms, Proceedings of the 6th ACM/IEEE-CS Joint Conference on Digital Libraries. JCDL ‘06., 286
  167. Jurafsky, 2023, Speech and Language Processing: An Introduction to Natural Language Processing, Computational Linguistics, and Speech Recognition, 3nd Edition draft
  168. Kanoulas, 2011, Evaluating Multi-Query Sessions, Proceedings of the 34th International ACM SIGIR Conference on Research and Development in Information Retrieval. SIGIR ‘11., 1053, 10.1145/2009916.2010056 · DOI
  169. Karmaker, 2020, Empirical Analysis of Impact of Query-Specific Customization of NDCG: A Case-Study with Learning-to-Rank Methods, Proceedings of the 29th ACM International Conference on Information & Knowledge Management. CIKM ‘20., 3281
  170. Käser, 2023, Simulated Learners in Educational Technology: A Systematic Literature Review and a Turing-like Test, International Journal of Artificial Intelligence in Education, 1
  171. Kelly, 2009, Methods for Evaluating Interactive Information Retrieval Systems with Users, Foundations and Trends in Information Retrieval., 3, 1, 10.1561/1500000012 · DOI
  172. Keskustalo, 2006, The Effects of Relevance Feedback Quality and Quantity in Interactive Relevance Feedback: A Simulation Based on User Modeling, Proceedings of the 28th European Conference on Advances in Information Retrieval. ECIR ‘06., 191
  173. Keskustalo, 2008, Evaluating the Effectiveness of Relevance Feedback Based on a User Simulation Model: Effects of a User Scenario on Cumulated Gain Value, Information Retrieval., 11, 209, 10.1007/s10791-007-9043-7 · DOI
  174. Keskustalo, 2009, Test Collection-Based IR Evaluation Needs Extension toward Sessions — A Case of Extremely Short Queries, Proceedings of the 5th Asia Information Retrieval Symposium on Information Retrieval Technology. AIRS ‘09., 63, 10.1007/978-3-642-04769-5_6 · DOI
  175. Khashabi, 2020, UNIFIEDQA: Crossing Format Boundaries with a Single QA System, Findings of the Association for Computational Linguistics: EMNLP 2020. EMNLP ‘20., 1896
  176. Kiesel, 2024, Simulating Follow-up Questions in Conversational Search, Proceedings of the 46th European Conference on IR Research. ECIR ‘24., 382
  177. Kim, 2021, NeuralWOZ: Learning to Collect Task-Oriented Dialogue via Model-Based Simulation, Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). ACL ‘21., 3704
  178. Kim, 2022, A Multi-Task Based Neural Model to Simulate Users in Goal Oriented Dialogue Systems, Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval. SIGIR ‘22., 2115, 10.1145/3477495.3531814 · DOI
  179. Kincaid, 1975, Derivation of New Readability Formulas (Automated Readability Index, Fog Count and Flesch Reading Ease Formula) for Navy Enlisted Personnel, Tech. rep. Institute for Simulation and Training
  180. Klöckner, 2004, Depth-and Breadth-First Processing of Search Result Lists, CHI ‘04 Extended Abstracts on Human Factors in Computing Systems. CHI EA ‘04.
  181. Koren, 2015, Advances in Collaborative Filtering, Recommender Systems Handbook., 77, 10.1007/978-1-4899-7637-6_3 · DOI
  182. Kostric, 2022, DAGFiNN: A Conversational Conference Assistant, RecSys ‘22., 628
  183. Kraft, 1979, Stopping Rules and their Effect on Expected Search Length, Information Processing & Management., 15, 47, 10.1016/0306-4573(79)90007-4 · DOI
  184. Krauth, 2020, Do Offline Metrics Predict Online Performance in Recommender Systems?, arXiv: 2011.07931 [cs.LG] · arXiv
  185. Krebs, 1973, Behavioral Aspects of Predation, Perspectives in Ethology., 73, 10.1007/978-1-4615-7569-6_3 · DOI
  186. Krebs, 1974, Hunting by Expectation or Optimal Foraging? A Study of Patch use by Chickadees, Animal Behaviour., 22, 953, 10.1016/0003-3472(74)90018-9 · DOI
  187. Kreyssig, 2018, Neural User Simulation for Corpus-based Policy Optimisation of Spoken Dialogue Systems, Proceedings of the 19th Annual SIG-dial Meeting on Discourse and Dialogue. SIGDIAL ‘18., 60, 10.18653/v1/W18-5007 · DOI
  188. Kuhlthau, 1991, Inside the Search Process: Information Seeking from the User's Perspective., Journal of the American Society for Information Science., 42, 361, 10.1002/(SICI)1097-4571(199106)42:5<361::AID-ASI6>3.0.CO;2-# · DOI
  189. Kuhlthau, 1988, Developing a Model of the Library Search Process: Cognitive and Affective Aspects, RQ., 28, 232
  190. Kunkel, 2017, A 3D Item Space Vi-sualization for Presenting and Manipulating User Preferences in Collaborative Filtering, Proceedings of the 22nd International Conference on Intelligent User Interfaces. IUI ‘17., 3
  191. Labhishetty, 2023, Models and Evaluation of User Simulation in Information Retrieval, PhD thesis. University of Illinois at UrbanaChampaign
  192. Labhishetty, 2022, PRE: A Precision-Recall-Effort Optimization Framework for Query Simulation, Proceedings of the 2022 ACM SIGIR International Conference on Theory of Information Retrieval. ICTIR ‘22., 51, 10.1145/3539813.3545136 · DOI
  193. Labhishetty, 2022, RATE: A Reliability-Aware Tester-Based Evaluation Framework of User Simulators, Proceedings of the 44th European Conference on IR Research. ECIR ‘22., 336
  194. Labhishetty, 2021, An Exploration of Tester-Based Evaluation of User Simulators for Comparing Interactive Retrieval Systems, Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval. SIGIR ‘21., 1598, 10.1145/3404835.3463091 · DOI
  195. Labhishetty, 2020, A Cognitive User Model for E-Commerce Search, Proceedings of the Data Science for Retail and E-Commerce Workshop.
  196. Lagun, 2014, Effects of Task and Domain on Searcher Attention, Proceedings of the 37th International ACM SIGIR Conference on Research and Development in Information Retrieval. SIGIR ‘14., 1087, 10.1145/2600428.2609516 · DOI
  197. Lee, 2019, ConvLab: Multi-Domain End-to-End Dialog System Platform, Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: System Demonstrations. ACL ‘19., 64, 10.18653/v1/P19-3011 · DOI
  198. Leuski, 2000, Relevance and Reinforcement in Interactive Browsing, Proceedings of the Ninth International Conference on Information and Knowledge Management. CIKM ‘00., 119
  199. Levin, 2000, A Stochastic Model of Human-machine Interaction for Learning Dialog Strategies, IEEE Transactions on Speech and Audio Processing., 8, 11, 10.1109/89.817450 · DOI
  200. Li, 2009, Good Abandonment in Mobile and PC Internet Search, Proceedings of the 32nd International ACM SIGIR Conference on Research and Development in Information Retrieval. SIGIR ‘09., 43, 10.1145/1571941.1571951 · DOI

200 of 404 shown

Cited by

Sorted by citations

Metadata

Authors
Balog, K., & Zhai, C.
Affiliations
  • University of Stavanger (NO)
  • University of Illinois Urbana-Champaign (US)
ORCID
Venue
Foundations and Trends® in Information Retrieval
Publisher
Emerald
Published
2024-06-13
Volume
18
Issue
1-2
Pages
1-261
ISSN
  • 1554-0669 (ISSN-L)
  • 1554-0677
DOI
10.1561/1500000098
OpenAlex
W4399631598
Cited by
52
References
13
Citations by year
2026: 27 · 2025: 12 · 2024: 11
FWCI
13.10
Type
journal article
Language
en
Open access
Locations
  • Foundations and Trends® in Information Retrieval (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Funders
  • National Science Foundation
  • Norges Forskningsråd
  • Institute of Education Sciences
Topics
  • Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
  • Physical Sciences › Computer Science › Information Systems › Recommender Systems and Techniques
  • Physical Sciences › Computer Science › Information Systems › Web Data Mining and Analysis
Keywords
user simulation, information access systems, system evaluation, search engines, recommender systems, chatbots, user modeling, dialogue systems, user interaction modeling

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review