A Comparative Analysis of Human and Automatic Query Variants
The DOI matches its Crossref record; 11 of the 11 references it cites were found in a registry or an index.
Abstract
We present an in-depth comparative analysis of the effectiveness distributions of sets of human-created and automatically-created query variations used to represent the same information need. The automatic variations are generated using Bing's click graph. Experiments performed with TREC datasets show that using automatic variations for retrieval can result in similar effectiveness to that of using human variations, although the two types of variations can be appreciably different in several important respects --- e.g., their similarities and corresponding retrieved lists.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Citations
Cites
A similarity measure for indefinite rankings
Webber, W., Moffat, A., & Zobel, J. (2010) · ACM Transactions on Information Systems · doi:10.1145/1852102.1852106 · Cited by 781
Understanding Equivalence and Noninferiority Testing
Walker, E., & Nowacki, A. S. (2011) · Journal of General Internal Medicine · doi:10.1007/s11606-010-1513-8 · Cited by 751 · cited 1×
“TOST is commonly used in the medical community to test for statistical non-inferiority [10].”
MethodRandom walks on the click graph
Craswell, N., & Szummer, M. (2007) · Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval · doi:10.1145/1277741.1277784 · Cited by 441 · cited 3×
“Sheldon et al. [8] proposed a process to induce multiple query variations from a starting query or description using the randomwalk model originally described by Craswell and Szummer [6].”
MethodResultSelf-citationA general language model for information retrieval (poster abstract)
Song, F., & Croft, W. B. (1999) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/312624.312698 · Cited by 146 · cited 1×
“Across all experiments, Krovetz stemming is applied to queries and documents, and query likelihood model (QL) [9] (Dirichlet smoothed document language model, μ=2500) was used for retrieval.”
MethodBuilding simulated queries for known-item topics: an analysis using six european languages
Azzopardi, L., de Rijke, M., & Balog, K. (2007) · Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval · doi:10.1145/1277741.1277820 · Cited by 135
Bailey, P., Moffat, A., Scholer, F., & Thomas, P. (2016) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/2911451.2914671 · Cited by 86 · cited 1×
“[2], andTREC descriptions were for ROBUST to ensure that the queries being generated were directly comparable to the human-generated query sets.”
BackgroundLambdaMerge: merging the results of query reformulations
Sheldon, D. R., Shokouhi, M., Szummer, M., & Craswell, N. (2011) · ACM International Conference on Web Search and Data Mining (WSDM) · doi:10.1145/1935826.1935930 · Cited by 76 · cited 1×
“[8] proposed a process to induce multiple query variations from a starting query or description using the randomwalk model originally described by Craswell and Szummer [6].”
MethodSelf-citationRetrieval Consistency in the Presence of Query Variations
Bailey, P., Moffat, A., Scholer, F., & Thomas, P. (2017) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3077136.3080839 · Cited by 75 · cited 2×
“query, even when very few variants are available [3, 4].”
BackgroundResultRisk-Reward Trade-offs in Rank Fusion
Benham, R., & Culpepper, J. S. (2017) · Australasian Document Computing Symposium · doi:10.1145/3166072.3166084 · Cited by 40 · cited 2×
“query, even when very few variants are available [3, 4].”
BackgroundResultSelf-citationTowards Efficient and Effective Query Variant Generation
Benham, R., Culpepper, J. S., Gallagher, L., Lu, X., & Mackenzie, J. (2018) · RMIT Research Repository (RMIT University Library) · Cited by 11
Self-citation
Research works printed but not linked: 1
- P. Bailey , A. Moffat , F. Scholer , and P. Thomas . 2016. UQV100: A Test Collection with Query Variability . In Proc. SIGIR. 725--728 . P. Bailey, A. Moffat, F. Scholer, and P. Thomas. 2016. UQV100: A Test Collection with Query Variability. In Proc. SIGIR. 725--728.
Cited by
CC-News-En: A Large English News Corpus
Mackenzie, J., Benham, R., Petri, M., Trippas, J. R., Culpepper, J. S., & Moffat, A. (2020) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3340531.3412762 · Cited by 94
“com/newsir19/ the consistency [9] and risk [11] properties of retrieval models; the quality of automatic query generation approaches [32]; and new implementation options for efficient search on web corpora [14].”
BackgroundSelf-citationA Study of a Gain Based Approach for Query Aspects in Recall Oriented Tasks
Nunzio, G. M. D., & Faggioli, G. (2021) · Applied Sciences · doi:10.3390/app11199075 · Cited by 14
“Recent studies have shown how query reformulations automatically extracted from query logs can be as effective as those manually created by users [9].”
BackgroundValidating Simulations of User Query Variants
Breuer, T., Fuhr, N., & Schaer, P. (2022) · Lecture notes in computer science · doi:10.1007/978-3-030-99736-6_6 · Cited by 22
Generating Natural Language Queries for More Effective Systematic Review Screening Prioritisation
Wang, S., Scells, H., Koopman, B., Potthast, M., & Zuccon, G. (2023) · Annual International ACM SIGIR Conference on Research and Development in Information Retrieval in the Asia Pacific Region (SIGIR-AP) · doi:10.1145/3624918.3625322 · Cited by 22
“Similar to the ChatGPT-generated queries, those from Alpaca also achieve higher effectiveness when fused with the results from the Boolean query.”
ResultChakraborty, A., Ganguly, D., & Conlan, O. (2020) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3340531.3412032 · Cited by 13
“effectiveness corresponding to the underlying information need of the original query [19].”
BackgroundOtero, D., Parapar, J., & Barreiro, Á. (2021) · ACM Symposium on Applied Computing · doi:10.1145/3412841.3441947 · Cited by 8
CODEC: Complex Document and Entity Collection
Mackie, I., Owoicho, P., Gemmell, C., Fischer, S., MacAvaney, S., & Dalton, J. (2022) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3477495.3531712 · Cited by 17
“Liu et al. [22] show the benefits of human and automatic query reformulations on document ranking systems, with human reformu-lations being the most effective.”
BackgroundModality Effects When Simulating User Querying Tasks
Mackenzie, J., & Moffat, A. (2021) · ACM SIGIR International Conference on Theory of Information Retrieval (ICTIR) · doi:10.1145/3471158.3472244 · Cited by 3
“This computationmeasures the intra-topic similarity [18], and provides a measure of similarity of the set of query variations for each topic.”
BackgroundMethodInfluentialStrong natural language query generation
Liu, B., Lu, X., & Culpepper, J. S. (2021) · Information Retrieval · doi:10.1007/s10791-021-09395-3 · Cited by 3
“For example, query log mining (Boldi et al. 2008; Sheldon et al. 2011; Liu et al. 2019) or crowdsourcing (Bailey et al. 2016).”
BackgroundSelf-citationDifferent Keystrokes for Different Folks
Benham, R., Mackenzie, J., Culpepper, J. S., & Moffat, A. (2021) · Conference on Human Information Interaction and Retrieval · doi:10.1145/3406522.3446054 · Cited by 2
“…understood since the early 1990s [4], UQV100 inspired a renewed interest in rank fusion [3, 5, 7, 9], automated query generation [6], and failure analyses, with the common goal of better understanding why human and machine-generated queries can have very different performance characteristics [15].”
BackgroundSelf-citationGenerative Agents Navigating Digital Libraries
Zerhoudi, S., & Granitzer, M. (2025) · Lecture notes in computer science · doi:10.1007/978-981-96-0865-2_14 · Cited by 1
“𝜏 quantifies shared keywords between paired queries using Jaccard similarity [19], indicating thematic similarity and consistent search intent.”
Classifying Term Variants in Query Formulation
Onq, N. A., Sanderson, M., & Scholer, F. (2025) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3726302.3729924 · Cited by 4
Cluster-Based Document Retrieval with Multiple Queries
Bernstein, K., Raiber, F., Kurland, O., & Culpepper, J. S. (2020) · ACM SIGIR International Conference on Theory of Information Retrieval (ICTIR) · doi:10.1145/3409256.3409825 · Cited by 1
“It was recently shown that query variations automatically selected from a query log of a commercial search engine can be, on average, as effective as human curated query variations [26].”
BackgroundSelf-citationUnderstanding and Modeling Heterogeneous Search Behavior
Onq, N. A., Ma, C., Sanderson, M., & Scholer, F. (2026) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3805712.3809618 · Cited by 2
Generative Agents Navigating Digital Libraries
Zerhoudi, S., & Granitzer, M. (2026) · arXiv · Cited by 1
“𝜏 quantifies shared keywords between paired queries using Jaccard similarity [19], indicating thematic similarity and consistent search intent.”
Metadata
- Authors
- Liu, B., Craswell, N., Lu, X., Kurland, O., & Culpepper, J. S.
- Affiliations
- RMIT University (AU)
- Microsoft (United States) (US)
- Technion – Israel Institute of Technology (IL)
- Venue
- Proceedings of the 2019 ACM SIGIR International Conference on Theory of Information Retrieval
- Publisher
- ACM
- Published
- 2019-09-26
- Pages
- 47-50
- OpenAlex
- W2975063679
- Cited by
- 14
- References
- 10
- Citations by year
- 2026: 1 · 2025: 1 · 2024: 1 · 2023: 1 · 2022: 2 · 2021: 5 · 2020: 3
- FWCI
- 1.99
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- ACM SIGIR International Conference on Theory of Information Retrieval (ICTIR) (publishedVersion)
- RMIT Research Repository (RMIT University Library) (submittedVersion)
- Figshare (submittedVersion)
- Funders
- Israel Science Foundation (1136/17)
- Australian Research Council (DP170102231, DP190101113)
- Topics
- Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
- Physical Sciences › Mathematics › Theoretical Computer Science › Algorithms and Data Compression
- Physical Sciences › Computer Science › Information Systems › Recommender Systems and Techniques
- Keywords
- automatic query generation, Bing, retrieval effectiveness, query expansion
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.