The Impact of Query Refinement on Systematic Review Literature Search: A Query Log Analysis
Open accessThe DOI matches its Crossref record; 20 of the 20 references it cites were found in a registry or an index.
Abstract
The creation of high-quality medical systematic reviews requires the development of a complex Boolean query to retrieve medical literature. An effective query in this context is critical, as it determines how many documents are to be assessed for inclusion in the resulting systematic review, as all retrieved documents must be screened. Therefore an effective query must balance a reasonable assessment workload with an estimate for how many relevant documents exist for a given topic. Getting this balance correct is naturally a difficult challenge, and there is a certain level of intuition involved in how a query should be formulated and refined. This paper reveals such intuitions and behaviours by analysing the query logs of a specialised tool developed to assist expert searchers in refining complex Boolean queries. These query logs contain unique information that permits a deeper understanding of user behaviour than previous studies. The approximately 6,000 queries collected over one year are available for further analysis at https://github.com/ielab/searchrefiner-logs-collection.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The text we found for this paper looks like another document, so the self-check wasn’t run.
Citations
Cites
Cochrane Handbook for Systematic Reviews of Interventions
Tarsilla, M. (2010) · Journal of MultiDisciplinary Evaluation · doi:10.56645/jmde.v6i14.284 · Cited by 5,178
A full systematic review was completed in 2 weeks using automation tools: a case study
Clark, J., Glasziou, P., Del Mar, C., Bannach-Brown, A., Stehlik, P., & Scott, A. M. (2020) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2020.01.008 · Cited by 687
Self-citationClark, J. M., Sanders, S., Carter, M., Honeyman, D., Cleo, G., Auld, Y., Booth, D., Condron, P., Dalais, C., Bateup, S., et al. (2020) · Journal of the Medical Library Association · doi:10.5195/jmla.2020.834 · Cited by 601 · cited 1×
“Due to the complexity of query formulation, and indeed refinement, of complex Boolean queries for systematic review literature search, tools for assisting users in these processes are not uncommon in this space [4, 14, 17, 20].”
BackgroundSelf-citationRethlefsen, M. L., Farrell, A. M., Osterhaus Trzasko, L. C., & Brigham, T. J. (2015) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2014.11.025 · Cited by 331 · cited 1×
“Although there have been several studies that have retrospectively analysed search strategies for systematic review literature search [5, 16], there are no studies that analyse query formulation and refinement prospectively (i.”
BackgroundA Day in the Life of PubMed: Analysis of a Typical Day's Query Log
Herskovic, J. R., Tanaka, L. Y., Hersh, W., & Bernstam, E. V. (2007) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m2191 · Cited by 170 · cited 1×
“These studies range from attempting to understand overall user behaviour [9, 10], to understanding search behaviours of experienced versus novice users [27], or even exploiting the query logs for suggestions [15].”
BackgroundErrors in search strategies were identified by type and frequency
Sampson, M., & McGowan, J. (2006) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2006.01.007 · Cited by 142 · cited 1×
“These automation tools may be able to alleviate some of the errors commonly found in these queries, such as spelling mistakes, irrelevant terms, or incorrect logical operators [5, 18], and lowering the time to create a review [3].”
BackgroundAutomatic boolean query suggestion for professional search
Kim, Y., Seo, J., & Croft, W. B. (2011) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/2009916.2010026 · Cited by 90 · cited 1×
“Despite such recommendations, the formulation of queries for this task, like many others, is openended, and there have been a variety of techniques for formulating searches [2, 8], and even methods that automatically formulate and refine queries [11, 19, 21, 22].”
MethodA large-scale analysis of query logs for assessing personalization opportunities
Wedig, S., & Madani, O. (2006) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/1150402.1150497 · Cited by 64 · cited 1×
“2 RELATEDWORK This paper attempts to understand the user behaviour for vastly different kinds of queries that are often seen in typical web search engine query logs [23, 26].”
BackgroundHausner, E., Guddat, C., Hermanns, T., Lampert, U., & Waffenschmidt, S. (2015) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2014.09.016 · Cited by 62 · cited 1×
“Seed studies are typically used to ‘validate’ a search [1, 2], but have also been used as a methodology for developing search strategies [7, 8, 22].”
MethodComparison of search strategies in systematic reviews of adverse effects to other systematic reviews
Golder, S., Loke, Y. K., & Zorzela, L. (2014) · Health Information & Libraries Journal · doi:10.1111/hir.12041 · Cited by 55 · cited 2×
“These automation tools may be able to alleviate some of the errors commonly found in these queries, such as spelling mistakes, irrelevant terms, or incorrect logical operators [5, 18], and lowering the time to create a review [3].”
BackgroundGenerating Better Queries for Systematic Reviews
Scells, H., & Zuccon, G. (2018) · The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval · doi:10.1145/3209978.3210020 · Cited by 52 · cited 2×
“Predicting when to stop refining is a crucial area of research in automatic query refinement [19, 21].”
BackgroundMethodSelf-citationsearchrefiner: A Query Visualisation and Understanding Tool for Systematic Reviews
Scells, H., & Zuccon, G. (2018) · Proceedings of the 27th ACM International Conference on Information and Knowledge Management · doi:10.1145/3269206.3269215 · Cited by 50 · cited 3×
“This paper seeks to analyse and understand the behaviour of users refining complex Boolean queries for systematic review literature search by way of interrogating the query logs of searchrefiner [20].”
BackgroundMethodSelf-citationYoo, I., & Mosa, A. S. M. (2015) · JMIR Medical Informatics · doi:10.2196/medinform.3740 · Cited by 39 · cited 1×
“These studies range from attempting to understand overall user behaviour [9, 10], to understanding search behaviours of experienced versus novice users [27], or even exploiting the query logs for suggestions [15].”
BackgroundAutomatic Boolean Query Refinement for Systematic Review Literature Search
Scells, H., Zuccon, G., & Koopman, B. (2019) · The World Wide Web Conference · doi:10.1145/3308558.3313544 · Cited by 38 · cited 3×
“Predicting when to stop refining is a crucial area of research in automatic query refinement [19, 21].”
BackgroundMethodSelf-citationSharing contrasting experiences at the U.S./U.K Exchange.
Rr, L. (2001) · PubMed · doi:10.5555/2491748.2491769 · Cited by 35
Systematic Reviewing: Introduction, Locating Studies and Data Abstraction
Clark, J. M. (2013) · Springer series on epidemiology and public health · doi:10.1007/978-3-642-37131-8_12 · Cited by 23
Self-citationSystematic Review Automation Tools for End-to-End Query Formulation
Li, H., Scells, H., & Zuccon, G. (2020) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3397271.3401402 · Cited by 21 · cited 1×
“Due to the complexity of query formulation, and indeed refinement, of complex Boolean queries for systematic review literature search, tools for assisting users in these processes are not uncommon in this space [4, 14, 17, 20].”
BackgroundSelf-citationWang, S., Scells, H., Clark, J., Koopman, B., & Zuccon, G. (2022) · Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval · doi:10.1145/3477495.3531748 · Cited by 21
Self-citationEnslaved to the Trapped Data: A Cognitive Work Analysis of Medical Systematic Reviews
Knight, I. A., Wilson, M. L., Brailsford, D. F., & Milić-Frayling, N. (2019) · Conference on Human Information Interaction and Retrieval · doi:10.1145/3295750.3298937 · Cited by 9 · cited 1×
“Boolean queries are needed for a variety of reasons, including compliance with policy or regulations, reproducibility of searches, and for transparency and understandability [12].”
Background
Research works printed but not linked: 7
- Elke Hausner , Siw Waffenschmidt , Thomas Kaiser , and Michael Simon . 2012. Routine Development of Objectively Derived Search Strategies. Systematic reviews 1, 1 ( 2012 ), 19. Elke Hausner, Siw Waffenschmidt, Thomas Kaiser, and Michael Simon. 2012. Routine Development of Objectively Derived Search Strategies. Systematic reviews 1, 1 (2012), 19.
- Rezarta Islamaj Dogan , G. Craig Murray , Aurélie Névéol , and Zhiyong Lu. 2009. Understanding PubMed User Search Behavior through Log Analysis. Database 2009 (Jan . 2009 ), bap018. Rezarta Islamaj Dogan, G. Craig Murray, Aurélie Névéol, and Zhiyong Lu. 2009. Understanding PubMed User Search Behavior through Log Analysis. Database 2009 (Jan. 2009), bap018.
- Grace E. Lee and Aixin Sun. 2018. Seed-Driven Document Ranking for Systematic Reviews in Evidence-Based Medicine . In Proceedings of the 41st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. 455--464 . Grace E. Lee and Aixin Sun. 2018. Seed-Driven Document Ranking for Systematic Reviews in Evidence-Based Medicine. In Proceedings of the 41st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. 455--464.
- Zhiyong Lu , W. John Wilbur , Johanna R McEntyre , Alexey Iskhakov , and Lee Szilagyi . 2009 . Finding Query Suggestions for PubMed. AMIA Annual Symposium Proceedings 2009 (2009), 396--400. Zhiyong Lu, W. John Wilbur, Johanna R McEntyre, Alexey Iskhakov, and Lee Szilagyi. 2009. Finding Query Suggestions for PubMed. AMIA Annual Symposium Proceedings 2009 (2009), 396--400.
- Tony Russell-Rose and Philip Gooch . 2018 . 2dSearch: A Visual Approach to Search Strategy Formulation . In Proceedings of the 1st Biennial Conference on Design of Experimental Search and Information Retrieval Systems. Tony Russell-Rose and Philip Gooch. 2018. 2dSearch: A Visual Approach to Search Strategy Formulation. In Proceedings of the 1st Biennial Conference on Design of Experimental Search and Information Retrieval Systems.
- Harrisen Scells , Guido Zuccon , and Bevan Koopman . 2020. A Comparison of Automatic Boolean Query Formulation for Systematic Reviews. Information Retrieval Journal ( 2020 ), 1--26. Harrisen Scells, Guido Zuccon, and Bevan Koopman. 2020. A Comparison of Automatic Boolean Query Formulation for Systematic Reviews. Information Retrieval Journal (2020), 1--26.
- Shuai Wang , Harrisen Scells , Ahmed Mourad , and Guido Zuccon . 2022. Seed- Driven Document Ranking for Systematic Reviews: A Reproducibility Study . In Advances in Information Retrieval (Lecture Notes in Computer Science), Matthias Hagen, Suzan Verberne, Craig Macdonald, Christin Seifert, Krisztian Balog, Kjetil Nørvåg , and Vinay Setty (Eds.). Springer International Publishing , Cham , 686--700. https://doi.org/10.1007/978--3-030--99736--6_46 Shuai Wang, Harrisen Scells, Ahmed Mourad, and Guido Zuccon. 2022. Seed- Driven Document Ranking for Systematic Reviews: A Reproducibility Study. In Advances in Information Retrieval (Lecture Notes in Computer Science), Matthias Hagen, Suzan Verberne, Craig Macdonald, Christin Seifert, Krisztian Balog, Kjetil Nørvåg, and Vinay Setty (Eds.). Springer International Publishing, Cham, 686--700. https://doi.org/10.1007/978--3-030--99736--6_46 · doi:10.1007/978--3-030--99736--6_46
Cited by
Sallam, M. (2023) · Healthcare · doi:10.3390/healthcare11060887 · Cited by 3,126
Lyu, Q., Tan, J., Zapadka, M. E., Ponnatapura, J., Niu, C., Myers, K. J., Wang, G., & Whitlow, C. T. (2023) · arXiv · doi:10.48550/arxiv.2303.09038 · Cited by 393
Lyu, Q., Tan, J., Zapadka, M. E., Ponnatapura, J., Niu, C., Myers, K. J., Wang, G., & Whitlow, C. T. (2023) · Visual Computing for Industry, Biomedicine, and Art · doi:10.1186/s42492-023-00136-5 · Cited by 367
Can ChatGPT Write a Good Boolean Query for Systematic Review Literature Search?
Wang, S., Scells, H., Koopman, B., & Zuccon, G. (2023) · Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval · doi:10.1145/3539618.3591703 · Cited by 275
Self-citationLeveraging ChatGPT to Aid Construction Hazard Recognition and Support Safety Education and Training
Uddin, S. M. J., Albert, A., Ovid, A., & Alsharef, A. (2023) · Sustainability · doi:10.3390/su15097121 · Cited by 158
Metadata
- Authors
- Scells, H., Forbes, C., Clark, J., Koopman, B., & Zuccon, G.
- Affiliations
- The University of Queensland (AU)
- Bond University (AU)
- Commonwealth Scientific and Industrial Research Organisation (AU)
- Venue
- Proceedings of the 2022 ACM SIGIR International Conference on Theory of Information Retrieval
- Publisher
- ACM
- Published
- 2022-08-23
- Pages
- 34-42
- OpenAlex
- W4293248820
- Cited by
- 12
- References
- 20
- Citations by year
- 2026: 1 · 2025: 1 · 2024: 2 · 2023: 8
- FWCI
- 0.35
- Type
- journal article
- Language
- en
- Open access
- green
- research.bond.edu.au
- Locations
- ACM SIGIR International Conference on Theory of Information Retrieval (ICTIR) (publishedVersion)
- Bond University Research Portal (Bond University) (publishedVersion, other-oa)
- Funders
- Australian Research Council (DP210104043)
- Topics
- Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Library and Information Sciences › scientometrics and bibliometrics research
- Keywords
- query refinement, Boolean queries, literature searching, systematic reviews, query logs, user behavior, search tools, medical literature searching
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.