CoChat

Assessing the performance of methodological search filters to improve the efficiency of evidence information retrieval: five literature reviews and a qualitative study

Open access
Lefebvre, Carol, Glanville, Julie, Beale, Sophie, Boachie, Charles, Duffy, Steven, Fraser, Cynthia, Harbour, Jenny, McCool, Rachael, Smith, Lynne · 2017 · Health Technology Assessment

The DOI matches its Crossref record; 129 of the 130 references it cites were found in a registry or an index, and 1 are not checked yet.

Abstract

Background: Effective study identification is essential for conducting health research, developing clinical guidance and health policy and supporting health-care decision-making. Methodological search filters (combinations of search terms to capture a specific study design) can assist in searching to achieve this. Objectives: This project investigated the methods used to assess the performance of methodological search filters, the information that searchers require when choosing search filters and how that information could be better provided. Methods: Five literature reviews were undertaken in 2010/11: search filter development and testing; comparison of search filters; decision-making in choosing search filters; diagnostic test accuracy (DTA) study methods; and decision-making in choosing diagnostic tests. We conducted interviews and a questionnaire with experienced searchers to learn what information assists in the choice of search filters and how filters are used. These investigations informed the development of various approaches to gathering and reporting search filter performance data. We acknowledge that there has been a regrettable delay between carrying out the project, including the searches, and the publication of this report, because of serious illness of the principal investigator. Results: The development of filters most frequently involved using a reference standard derived from hand-searching journals. Most filters were validated internally only. Reporting of methods was generally poor. Sensitivity, precision and specificity were the most commonly reported performance measures and were presented in tables. Aspects of DTA study methods are applicable to search filters, particularly in the development of the reference standard. There is limited evidence on how clinicians choose between diagnostic tests. No published literature was found on how searchers select filters. Interviewing and questioning searchers via a questionnaire found that filters were not appropriate for all tasks but were predominantly used to reduce large numbers of retrieved records and to introduce focus. The Inter Technology Appraisal Support Collaboration (InterTASC) Information Specialists’ Sub-Group (ISSG) Search Filters Resource was most frequently mentioned by both groups as the resource consulted to select a filter. Randomised controlled trial (RCT) and systematic review filters, in particular the Cochrane RCT and the McMaster Hedges filters, were most frequently mentioned. The majority indicated that they used different filters depending on the requirement for sensitivity or precision. Over half of the respondents used the filters available in databases. Interviewees used various approaches when using and adapting search filters. Respondents suggested that the main factors that would make choosing a filter easier were the availability of critical appraisals and more detailed performance information. Provenance and having the filter available in a central storage location were also important. Limitations: The questionnaire could have been shorter and could have included more multiple choice questions, and the reviews of filter performance focused on only four study designs. Conclusions: Search filter studies should use a representative reference standard and explicitly report methods and results. Performance measures should be presented systematically and clearly. Searchers find filters useful in certain circumstances but expressed a need for more user-friendly performance information to aid filter choice. We suggest approaches to use, adapt and report search filter performance. Future work could include research around search filters and performance measures for study designs not addressed here, exploration of alternative methods of displaying performance results and numerical synthesis of performance comparison results. Funding: The National Institute for Health Research (NIHR) Health Technology Assessment programme and Medical Research Council–NIHR Methodology Research Programme (grant number G0901496).

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.3310/hta21690

Cites

129 of 130 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Research works printed but not linked: 16

  1. Glanville, 2008, ISSG Search Filter Resource
  2. Glanville, 2009, Development and Testing of Search Filters to Identify Economic Evaluations in MEDLINE and EMBASE
  3. Bossuyt, 2008, Cochrane Handbook of Systematic Reviews of Diagnostic Test Accuracy Version 0.4
  4. Food and Drug Administration, 2007, Guidance for Industry and Staff: Statistical Guidance on Reporting Results from Studies Evaluating Diagnostic Tests
  5. Medical Services Advisory Committee, 2005, Guidelines for the Assessment of Diagnostic Technologies
  6. Belgian Health Care Knowledge Centre, 2005, HTA: Molecular Diagnostics in Belgium
  7. Agency for Healthcare Research and Quality, 2011, Comprehensive Overview of the Methods and Reporting of Meta-analyses of Test Accuracy
  8. Belgian Health Care Knowledge C, 2007, Search for Evidence and Critical Appraisal: Health Technology Assessment
  9. National Institute for Health and Care Excellence, 2010, Interim Methods Statement: Centre for Health Technology Evaluation, Diagnostics Assessment Programme
  10. UK National Screening Committee, 2011, Criteria for Appraising the Viability, Effectiveness and Appropriateness of a Screening Programme
  11. US Preventive Services Task Force, 2008, Procedure Manual
  12. Australian Population Health Development Principal Committee Screening Subcommittee, 2008, Population Based Screening Framework
  13. World Health Organization, 2011, Screening for Various Cancers
  14. Lefebvre, 2011, Cochrane Handbook for Systematic Reviews of Interventions
  15. Noyes, 2011, Cochrane Handbook for Systematic Reviews of Interventions
  16. McGowan, 2016, PRESS – Peer Review of Electronic Search Strategies: 2015 Guideline Explanation and Elaboration (PRESS E&E)

Cited by

See all 52

Metadata

Authors
Lefebvre, C., Glanville, J., Beale, S., Boachie, C., Duffy, S., Fraser, C., Harbour, J., McCool, R., & Smith, L.
Affiliations
  • Cochrane (GB)
  • York Health Economics Consortium, York, UK
  • University of Aberdeen (GB)
  • Healthcare Improvement Scotland (GB)
Corresponding author
Lefebvre, Carol
Venue
Health Technology Assessment
Published
2017-11
Volume
21
Issue
69
Pages
1-148
ISSN
  • 1366-5278 (ISSN-L)
  • 2046-4924
OpenAlex
W2774248142
Cited by
49
References
130
Citations by year
2026: 4 · 2025: 6 · 2024: 3 · 2023: 6 · 2022: 3 · 2021: 12 · 2020: 6 · 2019: 6 · 2018: 3
FWCI
3.67
Type
journal article
Language
en
Open access
Locations
  • Health Technology Assessment (publishedVersion, cc-by)
  • PubMed Central (publishedVersion)
  • PubMed (publishedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
Funders
  • Health Technology Assessment programme (06/92/58)
  • Medical Research Council
Topics
  • Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
  • Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
  • Health Sciences › Health Professions › General Health Professions › Health Sciences Research and Education
Keywords
reference materials, diagnostic accuracy, sensitivity and specificity, evidence retrieval, information professionals, critical appraisal
MeSH terms
Humans, Surveys and Questionnaires, Review Literature as Topic, Technology Assessment, Biomedical, Databases, Bibliographic, Information Storage and Retrieval, Qualitative Research, Search Engine

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.