A Comparative Analysis of Human and Automatic Query Variants
The DOI matches its Crossref record; 10 of the 10 references it cites were found in a registry or an index.
Abstract
We present an in-depth comparative analysis of the effectiveness distributions of sets of human-created and automatically-created query variations used to represent the same information need. The automatic variations are generated using Bing's click graph. Experiments performed with TREC datasets show that using automatic variations for retrieval can result in similar effectiveness to that of using human variations, although the two types of variations can be appreciably different in several important respects --- e.g., their similarities and corresponding retrieved lists.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Citations
Cites
A similarity measure for indefinite rankings
Webber, W., Moffat, A., & Zobel, J. (2010) · ACM Transactions on Information Systems · doi:10.1145/1852102.1852106 · Cited by 781
Understanding Equivalence and Noninferiority Testing
Walker, E., & Nowacki, A. S. (2011) · Journal of General Internal Medicine · doi:10.1007/s11606-010-1513-8 · Cited by 751 · cited 1×
“TOST is commonly used in the medical community to test for statistical non-inferiority [10].”
MethodRandom walks on the click graph
Craswell, N., & Szummer, M. (2007) · Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval · doi:10.1145/1277741.1277784 · Cited by 441 · cited 3×
“Sheldon et al. [8] proposed a process to induce multiple query variations from a starting query or description using the randomwalk model originally described by Craswell and Szummer [6].”
MethodResultSelf-citationA general language model for information retrieval (poster abstract)
Song, F., & Croft, W. B. (1999) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/312624.312698 · Cited by 146 · cited 1×
“Across all experiments, Krovetz stemming is applied to queries and documents, and query likelihood model (QL) [9] (Dirichlet smoothed document language model, μ=2500) was used for retrieval.”
MethodBailey, P., Moffat, A., Scholer, F., & Thomas, P. (2016) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/2911451.2914671 · Cited by 86 · cited 1×
“[2], andTREC descriptions were for ROBUST to ensure that the queries being generated were directly comparable to the human-generated query sets.”
BackgroundLambdaMerge: merging the results of query reformulations
Sheldon, D. R., Shokouhi, M., Szummer, M., & Craswell, N. (2011) · ACM International Conference on Web Search and Data Mining (WSDM) · doi:10.1145/1935826.1935930 · Cited by 76 · cited 1×
“[8] proposed a process to induce multiple query variations from a starting query or description using the randomwalk model originally described by Craswell and Szummer [6].”
MethodSelf-citationRetrieval Consistency in the Presence of Query Variations
Bailey, P., Moffat, A., Scholer, F., & Thomas, P. (2017) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3077136.3080839 · Cited by 75 · cited 2×
“query, even when very few variants are available [3, 4].”
BackgroundResultRisk-Reward Trade-offs in Rank Fusion
Benham, R., & Culpepper, J. S. (2017) · Australasian Document Computing Symposium · doi:10.1145/3166072.3166084 · Cited by 40 · cited 2×
“query, even when very few variants are available [3, 4].”
BackgroundResultSelf-citationTowards Efficient and Effective Query Variant Generation
Benham, R., Culpepper, J. S., Gallagher, L., Lu, X., & Mackenzie, J. (2018) · RMIT Research Repository (RMIT University Library) · Cited by 11
Self-citationThe Limits of Retrieval Effectiveness
Cummins, R., Lalmas, M., & O’Riordan, C. (2011) · Lecture notes in computer science · doi:10.1007/978-3-642-20161-5_27 · Cited by 6
Research works printed but not linked: 2
- L. Azzopardi , M. de Rijke , and K. Balog . 2007. Building simulated queries for knownitem topics: an analysis using six european languages . In Proc. SIGIR. 455--462 . L. Azzopardi, M. de Rijke, and K. Balog. 2007. Building simulated queries for knownitem topics: an analysis using six european languages. In Proc. SIGIR. 455--462.
- P. Bailey , A. Moffat , F. Scholer , and P. Thomas . 2016. UQV100: A Test Collection with Query Variability . In Proc. SIGIR. 725--728 . P. Bailey, A. Moffat, F. Scholer, and P. Thomas. 2016. UQV100: A Test Collection with Query Variability. In Proc. SIGIR. 725--728.
Cited by
CC-News-En: A Large English News Corpus
Mackenzie, J., Benham, R., Petri, M., Trippas, J. R., Culpepper, J. S., & Moffat, A. (2020) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3340531.3412762 · Cited by 94
“com/newsir19/ the consistency [9] and risk [11] properties of retrieval models; the quality of automatic query generation approaches [32]; and new implementation options for efficient search on web corpora [14].”
BackgroundSelf-citationA Study of a Gain Based Approach for Query Aspects in Recall Oriented Tasks
Nunzio, G. M. D., & Faggioli, G. (2021) · Applied Sciences · doi:10.3390/app11199075 · Cited by 14
“Recent studies have shown how query reformulations automatically extracted from query logs can be as effective as those manually created by users [9].”
BackgroundValidating Simulations of User Query Variants
Breuer, T., Fuhr, N., & Schaer, P. (2022) · Lecture notes in computer science · doi:10.1007/978-3-030-99736-6_6 · Cited by 22
Generating Natural Language Queries for More Effective Systematic Review Screening Prioritisation
Wang, S., Scells, H., Koopman, B., Potthast, M., & Zuccon, G. (2023) · Annual International ACM SIGIR Conference on Research and Development in Information Retrieval in the Asia Pacific Region (SIGIR-AP) · doi:10.1145/3624918.3625322 · Cited by 22
“Similar to the ChatGPT-generated queries, those from Alpaca also achieve higher effectiveness when fused with the results from the Boolean query.”
ResultChakraborty, A., Ganguly, D., & Conlan, O. (2020) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3340531.3412032 · Cited by 13
“effectiveness corresponding to the underlying information need of the original query [19].”
Background
Metadata
- Authors
- Liu, B., Craswell, N., Lu, X., Kurland, O., & Culpepper, J. S.
- Affiliations
- RMIT University (AU)
- Microsoft (United States) (US)
- Technion – Israel Institute of Technology (IL)
- Venue
- Proceedings of the 2019 ACM SIGIR International Conference on Theory of Information Retrieval
- Publisher
- ACM
- Published
- 2019-09-26
- Pages
- 47-50
- OpenAlex
- W2975063679
- Cited by
- 14
- References
- 10
- Citations by year
- 2026: 1 · 2025: 1 · 2024: 1 · 2023: 1 · 2022: 2 · 2021: 5 · 2020: 3
- FWCI
- 1.99
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- ACM SIGIR International Conference on Theory of Information Retrieval (ICTIR) (publishedVersion)
- RMIT Research Repository (RMIT University Library) (submittedVersion)
- Figshare (submittedVersion)
- Funders
- Israel Science Foundation (1136/17)
- Australian Research Council (DP170102231, DP190101113)
- Topics
- Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
- Physical Sciences › Mathematics › Theoretical Computer Science › Algorithms and Data Compression
- Physical Sciences › Computer Science › Information Systems › Recommender Systems and Techniques
- Keywords
- automatic query generation, Bing, retrieval effectiveness, query expansion
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: A Comparative Analysis of Human and Automatic Query Variants. https://app.cochat.ai/papers/10.1145/3341981.3344223
- The paper
- Liu, B., Craswell, N., Lu, X., Kurland, O., & Culpepper, J. S. (2019). A Comparative Analysis of Human and Automatic Query Variants. Proceedings of the 2019 ACM SIGIR International Conference on Theory of Information Retrieval. https://doi.org/10.1145/3341981.3344223
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.