CoChat

The Impact of Query Refinement on Systematic Review Literature Search: A Query Log Analysis

Open access
Scells, Harrisen, Forbes, Connor, Clark, Justin, Koopman, Bevan, Zuccon, Guido · 2022 · Proceedings of the 2022 ACM SIGIR International Conference on Theory of Information Retrieval

The DOI matches its Crossref record; 20 of the 20 references it cites were found in a registry or an index.

Abstract

The creation of high-quality medical systematic reviews requires the development of a complex Boolean query to retrieve medical literature. An effective query in this context is critical, as it determines how many documents are to be assessed for inclusion in the resulting systematic review, as all retrieved documents must be screened. Therefore an effective query must balance a reasonable assessment workload with an estimate for how many relevant documents exist for a given topic. Getting this balance correct is naturally a difficult challenge, and there is a certain level of intuition involved in how a query should be formulated and refined. This paper reveals such intuitions and behaviours by analysing the query logs of a specialised tool developed to assist expert searchers in refining complex Boolean queries. These query logs contain unique information that permits a deeper understanding of user behaviour than previous studies. The approximately 6,000 queries collected over one year are available for further analysis at https://github.com/ielab/searchrefiner-logs-collection.

Overview

Integrity signals

Registry

Crossref · 10.1145/3539813.3545143

Cites

20 of 20 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Verification

Claims the paper makes

The text we found for this paper looks like another document, so its claims and self-check weren’t run. The registry and citation checks still apply.

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The text we found for this paper looks like another document, so the self-check wasn’t run.

Citations

Cites

Research works printed but not linked: 7

  1. Elke Hausner , Siw Waffenschmidt , Thomas Kaiser , and Michael Simon . 2012. Routine Development of Objectively Derived Search Strategies. Systematic reviews 1, 1 ( 2012 ), 19. Elke Hausner, Siw Waffenschmidt, Thomas Kaiser, and Michael Simon. 2012. Routine Development of Objectively Derived Search Strategies. Systematic reviews 1, 1 (2012), 19.
  2. Rezarta Islamaj Dogan , G. Craig Murray , Aurélie Névéol , and Zhiyong Lu. 2009. Understanding PubMed User Search Behavior through Log Analysis. Database 2009 (Jan . 2009 ), bap018. Rezarta Islamaj Dogan, G. Craig Murray, Aurélie Névéol, and Zhiyong Lu. 2009. Understanding PubMed User Search Behavior through Log Analysis. Database 2009 (Jan. 2009), bap018.
  3. Grace E. Lee and Aixin Sun. 2018. Seed-Driven Document Ranking for Systematic Reviews in Evidence-Based Medicine . In Proceedings of the 41st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. 455--464 . Grace E. Lee and Aixin Sun. 2018. Seed-Driven Document Ranking for Systematic Reviews in Evidence-Based Medicine. In Proceedings of the 41st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. 455--464.
  4. Zhiyong Lu , W. John Wilbur , Johanna R McEntyre , Alexey Iskhakov , and Lee Szilagyi . 2009 . Finding Query Suggestions for PubMed. AMIA Annual Symposium Proceedings 2009 (2009), 396--400. Zhiyong Lu, W. John Wilbur, Johanna R McEntyre, Alexey Iskhakov, and Lee Szilagyi. 2009. Finding Query Suggestions for PubMed. AMIA Annual Symposium Proceedings 2009 (2009), 396--400.
  5. Tony Russell-Rose and Philip Gooch . 2018 . 2dSearch: A Visual Approach to Search Strategy Formulation . In Proceedings of the 1st Biennial Conference on Design of Experimental Search and Information Retrieval Systems. Tony Russell-Rose and Philip Gooch. 2018. 2dSearch: A Visual Approach to Search Strategy Formulation. In Proceedings of the 1st Biennial Conference on Design of Experimental Search and Information Retrieval Systems.
  6. Harrisen Scells , Guido Zuccon , and Bevan Koopman . 2020. A Comparison of Automatic Boolean Query Formulation for Systematic Reviews. Information Retrieval Journal ( 2020 ), 1--26. Harrisen Scells, Guido Zuccon, and Bevan Koopman. 2020. A Comparison of Automatic Boolean Query Formulation for Systematic Reviews. Information Retrieval Journal (2020), 1--26.
  7. Shuai Wang , Harrisen Scells , Ahmed Mourad , and Guido Zuccon . 2022. Seed- Driven Document Ranking for Systematic Reviews: A Reproducibility Study . In Advances in Information Retrieval (Lecture Notes in Computer Science), Matthias Hagen, Suzan Verberne, Craig Macdonald, Christin Seifert, Krisztian Balog, Kjetil Nørvåg , and Vinay Setty (Eds.). Springer International Publishing , Cham , 686--700. https://doi.org/10.1007/978--3-030--99736--6_46 Shuai Wang, Harrisen Scells, Ahmed Mourad, and Guido Zuccon. 2022. Seed- Driven Document Ranking for Systematic Reviews: A Reproducibility Study. In Advances in Information Retrieval (Lecture Notes in Computer Science), Matthias Hagen, Suzan Verberne, Craig Macdonald, Christin Seifert, Krisztian Balog, Kjetil Nørvåg, and Vinay Setty (Eds.). Springer International Publishing, Cham, 686--700. https://doi.org/10.1007/978--3-030--99736--6_46 · doi:10.1007/978--3-030--99736--6_46

Cited by

See all 13

Metadata

Authors
Scells, H., Forbes, C., Clark, J., Koopman, B., & Zuccon, G.
Affiliations
  • The University of Queensland (AU)
  • Bond University (AU)
  • Commonwealth Scientific and Industrial Research Organisation (AU)
Venue
Proceedings of the 2022 ACM SIGIR International Conference on Theory of Information Retrieval
Publisher
ACM
Published
2022-08-23
Pages
34-42
OpenAlex
W4293248820
Cited by
12
References
20
Citations by year
2026: 1 · 2025: 1 · 2024: 2 · 2023: 8
FWCI
0.35
Type
journal article
Language
en
Open access
Locations
  • ACM SIGIR International Conference on Theory of Information Retrieval (ICTIR) (publishedVersion)
  • Bond University Research Portal (Bond University) (publishedVersion, other-oa)
Funders
Australian Research Council (DP210104043)
Topics
  • Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
  • Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
  • Social Sciences › Library and Information Sciences › scientometrics and bibliometrics research
Keywords
query refinement, Boolean queries, literature searching, systematic reviews, query logs, user behavior, search tools, medical literature searching

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.