Assessing the performance of methodological search filters to improve the efficiency of evidence information retrieval: five literature reviews and a qualitative study
Open accessThe DOI matches its Crossref record; 129 of the 130 references it cites were found in a registry or an index, and 1 are not checked yet.
Abstract
Background: Effective study identification is essential for conducting health research, developing clinical guidance and health policy and supporting health-care decision-making. Methodological search filters (combinations of search terms to capture a specific study design) can assist in searching to achieve this. Objectives: This project investigated the methods used to assess the performance of methodological search filters, the information that searchers require when choosing search filters and how that information could be better provided. Methods: Five literature reviews were undertaken in 2010/11: search filter development and testing; comparison of search filters; decision-making in choosing search filters; diagnostic test accuracy (DTA) study methods; and decision-making in choosing diagnostic tests. We conducted interviews and a questionnaire with experienced searchers to learn what information assists in the choice of search filters and how filters are used. These investigations informed the development of various approaches to gathering and reporting search filter performance data. We acknowledge that there has been a regrettable delay between carrying out the project, including the searches, and the publication of this report, because of serious illness of the principal investigator. Results: The development of filters most frequently involved using a reference standard derived from hand-searching journals. Most filters were validated internally only. Reporting of methods was generally poor. Sensitivity, precision and specificity were the most commonly reported performance measures and were presented in tables. Aspects of DTA study methods are applicable to search filters, particularly in the development of the reference standard. There is limited evidence on how clinicians choose between diagnostic tests. No published literature was found on how searchers select filters. Interviewing and questioning searchers via a questionnaire found that filters were not appropriate for all tasks but were predominantly used to reduce large numbers of retrieved records and to introduce focus. The Inter Technology Appraisal Support Collaboration (InterTASC) Information Specialists’ Sub-Group (ISSG) Search Filters Resource was most frequently mentioned by both groups as the resource consulted to select a filter. Randomised controlled trial (RCT) and systematic review filters, in particular the Cochrane RCT and the McMaster Hedges filters, were most frequently mentioned. The majority indicated that they used different filters depending on the requirement for sensitivity or precision. Over half of the respondents used the filters available in databases. Interviewees used various approaches when using and adapting search filters. Respondents suggested that the main factors that would make choosing a filter easier were the availability of critical appraisals and more detailed performance information. Provenance and having the filter available in a central storage location were also important. Limitations: The questionnaire could have been shorter and could have included more multiple choice questions, and the reviews of filter performance focused on only four study designs. Conclusions: Search filter studies should use a representative reference standard and explicitly report methods and results. Performance measures should be presented systematically and clearly. Searchers find filters useful in certain circumstances but expressed a need for more user-friendly performance information to aid filter choice. We suggest approaches to use, adapt and report search filter performance. Future work could include research around search filters and performance measures for study designs not addressed here, exploration of alternative methods of displaying performance results and numerical synthesis of performance comparison results. Funding: The National Institute for Health Research (NIHR) Health Technology Assessment programme and Medical Research Council–NIHR Methodology Research Programme (grant number G0901496).
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“This enables readers to assess the quality of the filter(s) used, as most published search filters are validated and/or peer reviewed [106, 107].”
From PRISMA-S: an extension to the PRISMA Statement for Reporting Literature Searches in Systematic Reviews (2021) · cites this paper as [106]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The relationship between Precision-Recall and ROC curves
Davis, J., & Goadrich, M. (2006) · Proceedings of the 23rd international conference on Machine learning - ICML '06 · doi:10.1145/1143844.1143874 · Cited by 6,559
PRESS Peer Review of Electronic Search Strategies: 2015 Guideline Statement
McGowan, J., Sampson, M., Salzwedel, D. M., Cogo, E., Foerster, V., & Lefebvre, C. (2016) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2016.01.021 · Cited by 5,700
Self-citationCochrane Handbook for Systematic Reviews of Interventions
Tarsilla, M. (2010) · Journal of MultiDisciplinary Evaluation · doi:10.56645/jmde.v6i14.284 · Cited by 5,178
Whiting, P., Rutjes, A. W., Reitsma, J. B., Bossuyt, P. M., & Kleijnen, J. (2003) · BMC Medical Research Methodology · doi:10.1186/1471-2288-3-25 · Cited by 3,592
Reitsma, J. B., Glas, A. S., Rutjes, A. W. S., Scholten, R. J. P. M., Bossuyt, P. M., & Zwinderman, A. H. (2005) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2005.02.022 · Cited by 3,369
Systematic reviews: CRD's guidance for undertaking reviews in health care
Tacconelli, E. (2010) · The Lancet Infectious Diseases · doi:10.1016/s1473-3099(10)70065-7 · Cited by 3,245
Towards complete and accurate reporting of studies of diagnostic accuracy: The STARD Initiative.
Bossuyt, P. M., Reitsma, J. B., Bruns, D. E., Gatsonis, C. A., Glasziou, P. P., Irwig, L. M., Lijmer, J. G., Moher, D., Rennie, D., de Vet, H. C. W., et al. (2003) · Annals of Internal Medicine · doi:10.7326/0003-4819-138-1-200301070-00010 · Cited by 3,219
Towards complete and accurate reporting of studies of diagnostic accuracy: the STARD initiative
Bossuyt, P. M., Reitsma, J. B., Bruns, D. E., Gatsonis, C. A., Glasziou, P. P., Irwig, L. M., Lijmer, J. G., Moher, D., Rennie, D., & de Vet, H. C. W. (2003) · BMJ · doi:10.1136/bmj.326.7379.41 · Cited by 3,215
Deeks, J. J., Altman, D. G., & Bradburn, M. J. (2001) · Systematic Reviews in Health Care · doi:10.1002/9780470693926.ch15 · Cited by 2,163
Research works printed but not linked: 16
- Glanville, 2008, ISSG Search Filter Resource
- Glanville, 2009, Development and Testing of Search Filters to Identify Economic Evaluations in MEDLINE and EMBASE
- Bossuyt, 2008, Cochrane Handbook of Systematic Reviews of Diagnostic Test Accuracy Version 0.4
- Food and Drug Administration, 2007, Guidance for Industry and Staff: Statistical Guidance on Reporting Results from Studies Evaluating Diagnostic Tests
- Medical Services Advisory Committee, 2005, Guidelines for the Assessment of Diagnostic Technologies
- Belgian Health Care Knowledge Centre, 2005, HTA: Molecular Diagnostics in Belgium
- Agency for Healthcare Research and Quality, 2011, Comprehensive Overview of the Methods and Reporting of Meta-analyses of Test Accuracy
- Belgian Health Care Knowledge C, 2007, Search for Evidence and Critical Appraisal: Health Technology Assessment
- National Institute for Health and Care Excellence, 2010, Interim Methods Statement: Centre for Health Technology Evaluation, Diagnostics Assessment Programme
- UK National Screening Committee, 2011, Criteria for Appraising the Viability, Effectiveness and Appropriateness of a Screening Programme
- US Preventive Services Task Force, 2008, Procedure Manual
- Australian Population Health Development Principal Committee Screening Subcommittee, 2008, Population Based Screening Framework
- World Health Organization, 2011, Screening for Various Cancers
- Lefebvre, 2011, Cochrane Handbook for Systematic Reviews of Interventions
- Noyes, 2011, Cochrane Handbook for Systematic Reviews of Interventions
- McGowan, 2016, PRESS – Peer Review of Electronic Search Strategies: 2015 Guideline Explanation and Elaboration (PRESS E&E)
Cited by
Rethlefsen, M. L., Kirtley, S., Waffenschmidt, S., Ayala, A. P., Moher, D., Page, M. J., Koffel, J. B., PRISMA-S Group, Blunt, H., Brigham, T., et al. (2021) · Systematic Reviews · doi:10.1186/s13643-020-01542-z · Cited by 3,714
Self-citationRethlefsen, M. L., Kirtley, S., Waffenschmidt, S., Ayala, A. P., Moher, D., Page, M. J., & Koffel, J. (2019) · doi:10.31219/osf.io/sfc38 · Cited by 3,696
Rethlefsen, M. L., Kirtley, S., Waffenschmidt, S., Ayala, A. P., Moher, D., Page, M. J., Koffel, J. B., & Group, P.-S. (2021) · Journal of the Medical Library Association · doi:10.5195/jmla.2021.962 · Cited by 409
Point-of-care ultrasonography for diagnosing thoracoabdominal injuries in patients with blunt trauma
Stengel, D., Leisterer, J., Ferrada, P. A., Ekkernkamp, A., Mutze, S., & Hönning, A. (2018) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd012669.pub2 · Cited by 189
Prady, S. L., Uphoff, E., Power, M., & Golder, S. (2018) · BMC Medical Research Methodology · doi:10.1186/s12874-018-0567-x · Cited by 75
Search strategies to identify observational studies in MEDLINE and Embase
Li, L., Smith, H. E., Atun, R. A., & Car, L. T. (2019) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.mr000041.pub2 · Cited by 71
Which are the most sensitive search filters to identify randomized controlled trials in MEDLINE?
Glanville, J., Kotas, E., Featherstone, R. M., & Dooley, G. (2020) · Journal of the Medical Library Association JMLA · doi:10.5195/jmla.2020.912 · Cited by 62
Self-citationCalcium and vitamin D for increasing bone mineral density in premenopausal women
Méndez-Sánchez, L., Clark, P., Winzenberg, T. M., Tugwell, P., Correa-Burrows, P., & Costello, R. (2023) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd012664.pub2 · Cited by 55
Topic search filters: a systematic scoping review
Damarell, R. A., May, N., Hammond, S., Sladek, R. M., & Tieman, J. J. (2019) · Health Information & Libraries Journal · doi:10.1111/hir.12244 · Cited by 48
Development of an efficient search filter to retrieve systematic reviews from PubMed
Oliván, J. A. S., Cuenca, G. M., & Avilés, R. A. (2021) · Journal of the Medical Library Association JMLA · doi:10.5195/jmla.2021.1223 · Cited by 47
Waffenschmidt, S., Navarro, T., Hobson, N., Hausner, E., Sauerland, S., & Haynes, R. B. (2020) · Research Synthesis Methods · doi:10.1002/jrsm.1425 · Cited by 40
Established search filters may miss studies when identifying randomized controlled trials
Cooper, C., Varley-Campbell, J., & Carter, P. (2019) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2019.04.002 · Cited by 38
Avau, B., Remoortel, H. V., & Buck, E. D. J. D. (2021) · Journal of the Medical Library Association JMLA · doi:10.5195/jmla.2021.1219 · Cited by 36
Shrestha, T., Chi, C. V. Y., Cassarino, M., Foley, S., & Blasi, Z. D. (2025) · Environment International · doi:10.1016/j.envint.2024.109217 · Cited by 39
Zhang, J., Han, L., Shields, L. E., Tian, J., & Wang, J. (2019) · Medicine · doi:10.1097/md.0000000000018099 · Cited by 35
Herber, O., & Barroso, J. (2020) · Qualitative Research · doi:10.1177/1468794119862440 · Cited by 25
Sutton, A., & Campbell, F. (2022) · Research Synthesis Methods · doi:10.1002/jrsm.1552 · Cited by 32
Ayiku, L., Hudson, T., Glover, S., Walsh, N. E., Adams, R. L., Deane, J., & Finnegan, A. (2020) · International Journal of Technology Assessment in Health Care · doi:10.1017/s026646232000080x · Cited by 27
Morel, T., Nguyen‐Soenen, J., Thompson, W., & Fournier, J. (2022) · BMC Medical Research Methodology · doi:10.1186/s12874-022-01515-x · Cited by 18
Cooper, C., Garside, R., Varley‐Campbell, J., Talens‐Bou, J., Booth, A., & Britten, N. (2020) · Research Synthesis Methods · doi:10.1002/jrsm.1426 · Cited by 17
Search strategies (filters) to identify systematic reviews in MEDLINE and Embase
Liquitay, C. M. E., Garegnani, L., Garrote, V., Solà, I., & Franco, J. V. A. (2023) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.mr000054.pub2 · Cited by 23
Rosumeck, S., Wagner, M., Wallraf, S., & von Euler, U. S. (2020) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2020.09.031 · Cited by 14
Learning the Nuclear: Educational Tourism in (Post)Industrial Sites
Mažeikienė, N. (2021) · Peter Lang D eBooks · doi:10.3726/b18090 · Cited by 9
Hazelton, C., Todhunter‐Brown, A., Campbell, P., Thomson, K., Nicolson, D. J., McGill, K., Chung, C. S., Dorris, L., Gillespie, D. C., Hunter, S. M., et al. (2023) · Health Technology Assessment · doi:10.3310/wgjt3471 · Cited by 8
Validation of a search strategy for randomized clinical trials related to periodontitis
Lyrio, A. O., da Cruz, S. S., Gomes‐Filho, I. S., Silveira, V. S. S., Souza, E. S., Batista, J. E. T., Figueiredo, A. C. M. G., & Pereira, M. G. (2021) · Archives of Public Health · doi:10.1186/s13690-021-00560-0 · Cited by 8
Pachanov, A., Muente, C., Hirt, J., & Pieper, D. (2024) · Research Synthesis Methods · doi:10.1002/jrsm.1763 · Cited by 6
Morel, T., Nguyen‐Soenen, J., Thompson, W., & Fournier, J. (2023) · Health Information & Libraries Journal · doi:10.1111/hir.12516 · Cited by 5
A Productive Proposed Search Syntax for Health Disaster Preparedness Research
Rastegarfar, B., Ardalan, A., Nejat, S., Keshtkar, A., & Moradian, M. J. (2019) · Bulletin of Emergency and Trauma · doi:10.29252/beat-070201 · Cited by 5
Escrivá, L., Hessel, E. V. S., Gustafsson, S., van Spronsen, R., Svanberg, M., & Beronius, A. (2020) · Environment International · doi:10.1016/j.envint.2020.105828 · Cited by 5
Use of deprescribing search filters in systematic review search strategies: A case study
Morel, T., Heinrich, C. H., Zerah, L., Hurley, E., Christiaens, A., & Fournier, J. (2023) · Basic & Clinical Pharmacology & Toxicology · doi:10.1111/bcpt.13908 · Cited by 5
Dinh, C., Marshall, Z., Marsalis, S. E., Gagen, R., Kang, P., & Everhart, A. R. (2025) · JMIR Research Protocols · doi:10.2196/76055 · Cited by 2
Validated geographic search filters for bibliographic databases: a scoping review
Muente, C., Pachanov, A., Hirt, J., Ayiku, L., & Pieper, D. (2025) · JBI Evidence Synthesis · doi:10.11124/jbies-24-00395 · Cited by 2
Metzendorf, M., Monsef, I., Jones, K. L., Wieland, L. S., Janka, H., Liquitay, C. M. E., & Thomson, D. (2026) · Cochrane Evidence Synthesis and Methods · doi:10.1002/cesm.70105 · Cited by 2
Unger, S., Raak, C., & Ostermann, T. (2023) · SN Computer Science · doi:10.1007/s42979-023-02146-9 · Cited by 1
Cooper, C., Kaunelis, D., Varley‐Campbell, J., & Carter, P. (2019) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2019.09.008 · Cited by 1
Evaluating and optimizing the search process for systematic reviews
Mathisen, M. (2021) · NORA - Norwegian Open Research Archives · Cited by 5
Learning the Nuclear: Educational Tourism in (Post)Industrial Sites
Mažeikienė, N. (2021) · Peter Lang D eBooks · doi:10.3726/9783631847343.003.0003 · Cited by 2
Learning the Nuclear: Educational Tourism in (Post)Industrial Sites
Mažeikienė, N. (2021) · Peter Lang D eBooks · doi:10.3726/9783631847343.003.0008
Learning the Nuclear: Educational Tourism in (Post)Industrial Sites
Mažeikienė, N. (2021) · Peter Lang D eBooks · doi:10.3726/9783631847343.003.0002
Rethlefsen, M. L. (2024) · Research Publications (Maastricht University) · doi:10.26481/dis.20250121mr
Pachanov, A., Muente, C., Hirt, J., & Pieper, D. (2025) · Research Synthesis Methods · doi:10.1017/rsm.2025.10016
Getting started with search filters in primary care literature reviews
Morel, T., Granikov, V., Kulshreshtha, A., Young, R. A., & Fournier, J. (2025) · Family Practice · doi:10.1093/fampra/cmaf037
Geographic search filters: promising tools for contextualizing research
Muente, C., Pachanov, A., Hirt, J., Ayiku, L., & Pieper, D. (2025) · JBI Evidence Synthesis · doi:10.11124/jbies-25-00344
Exercise-based rehabilitation for ACL injuries: a systematic review and meta-analysis
Verhagen, A. P., Nasser, A. M., Jenkins, L. C., Monk, A. P., Saragiotto, B. T., & Stubbs, P. W. (2025) · Musculoskeletal Science and Practice · doi:10.1016/j.msksp.2025.103392
Lyrio, A. O., da Cruz, S. S., Gomes-Filho, I. S., Silveira, V. S. S., Souza, E. S., Batista, J. E. T., Figueiredo, A. C. M. G., & Pereira, M. G. (2021) · Figshare · doi:10.6084/m9.figshare.14369902
Rethlefsen, M., Kirtley, S., Waffenschmidt, S., Ayala, A. P., Moher, D., Page, M. J., & Koffel, J. B. (2021) · Figshare · doi:10.6084/m9.figshare.13646921
Morel, T., Nguyen‐Soenen, J., Thompson, W., & Fournier, J. (2022) · Figshare · doi:10.6084/m9.figshare.19424891
Why African American Men Diagnosed With Chronic Obstructive Pulmonary Disease Continue to Smoke
Allen, V. L. (2021) · ScholarWorks (Walden University)
Capturing financial experiences of cancer survivors: a scoping review and call for consensus
Park, E. R., Lydston, M., Durieux, N. M., Beauchemin, M., Badr, H., de Moor, J. S., Mitchell, C. O., Troubh, Z., Lopez, P. L. V., Halpern, M. T., et al. (2026) · Cancer Survivorship Research & Care · doi:10.1080/28352610.2026.2675187
Lin, Y.-C., Tsai, P.-H., Weng, Y.-H., Chiu, H.-L., Ma, C.-Y., & Chang, Y.-J. (2026) · Designing Interactive Systems Conference · doi:10.1145/3800645.3813043
Gagen, R., Marsalis, S. E., Dinh, C., Applejohn, M., Shah, R., Tong, O. X., Marshall, Z., & Everhart, A. R. (2026) · PLoS ONE · doi:10.1371/journal.pone.0351303
Faherty, M., Madden, R., Herlihy, B., Bowman, S., MacLullich, A. M. J., & Timmons, S. (2026) · BMC Medical Research Methodology · doi:10.1186/s12874-026-02941-x
Metadata
- Authors
- Lefebvre, C., Glanville, J., Beale, S., Boachie, C., Duffy, S., Fraser, C., Harbour, J., McCool, R., & Smith, L.
- Affiliations
- Cochrane (GB)
- York Health Economics Consortium, York, UK
- University of Aberdeen (GB)
- Healthcare Improvement Scotland (GB)
- Corresponding author
- Lefebvre, Carol
- ORCID
- Venue
- Health Technology Assessment
- Published
- 2017-11
- Volume
- 21
- Issue
- 69
- Pages
- 1-148
- ISSN
- 1366-5278 (ISSN-L)
- 2046-4924
- PMC
- PMC5723931
- OpenAlex
- W2774248142
- Cited by
- 49
- References
- 130
- Citations by year
- 2026: 4 · 2025: 6 · 2024: 3 · 2023: 6 · 2022: 3 · 2021: 12 · 2020: 6 · 2019: 6 · 2018: 3
- FWCI
- 3.67
- Type
- journal article
- Language
- en
- Open access
- diamond
- doi.org
- Locations
- Health Technology Assessment (publishedVersion, cc-by)
- PubMed Central (publishedVersion)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- Health Technology Assessment programme (06/92/58)
- Medical Research Council
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
- Health Sciences › Health Professions › General Health Professions › Health Sciences Research and Education
- Keywords
- reference materials, diagnostic accuracy, sensitivity and specificity, evidence retrieval, information professionals, critical appraisal
- MeSH terms
- Humans, Surveys and Questionnaires, Review Literature as Topic, Technology Assessment, Biomedical, Databases, Bibliographic, Information Storage and Retrieval, Qualitative Research, Search Engine
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.