An alternative to the hand searching gold standard: validating methodological search filters using relative recall
Open accessThe DOI matches its Crossref record; 9 of the 9 references it cites were found in a registry or an index.
Abstract
BACKGROUND: Search filters or hedges play an important role in evidence-based medicine but their development depends on the availability of a "gold standard"--a reference standard against which to establish the performance of the filter. We demonstrate the feasibility of using relative recall of included studies from multiple systematic reviews to validate methodological search filters as an alternative to validation against a gold standard formed through hand searching. METHODS: We identified 105 Cochrane reviews that used the Highly Sensitive Search Strategy (HSSS), included randomized or quasi-randomized controlled trials, and reported their included studies. We measured the ability of two published and one novel variant of the HSSS to retrieve the MEDLINE-index studies included in these reviews. RESULTS: The systematic reviews were comprehensive in their searches. 72% of included primary studies were indexed in MEDLINE. Relative recall of the three strategies ranged from .98 to .91 across all reviews and more comprehensive strategies showed higher recall. CONCLUSION: An approach using relative recall instead of a hand searching gold standard proved feasible and produced recall figures that were congruent with previously published figures for the HSSS. This technique would permit validation of a methodological filter using a collection of approximately 100 studies of the chosen design drawn from the included studies of multiple systematic reviews that used comprehensive search strategies.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Robinson, K. A., & Dickersin, K. (2002) · International Journal of Epidemiology · doi:10.1093/ije/31.1.150 · Cited by 833
Royle, P., & Milne, R. (2003) · International Journal of Technology Assessment in Health Care · doi:10.1017/s0266462303000552 · Cited by 693
Formulating Questions and Locating Primary Studies for Inclusion in Systematic Reviews
Counsell, C. (1997) · Annals of Internal Medicine · doi:10.7326/0003-4819-127-5-199709010-00008 · Cited by 505
Reducing Workload in Systematic Review Preparation Using Automated Citation Classification
Cohen, A. M., Hersh, W. R., Peterson, K., & Yen, P.-Y. (2006) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m1929 · Cited by 411
Developing a database of published reports of randomised clinical trials in pain research 1
Jadad, A. R., Carroll, D., Moore, A., & McQuay, H. (1996) · Pain · doi:10.1016/0304-3959(96)03033-3 · Cited by 271
Development of The Cochrane Collaboration's Central Register of Controlled Clinical Trials
Dickersin, K., Manheimer, E., Wieland, S., Robinson, K. A., Lefebvre, C., & McDonald, S. (2002) · Evaluation & the Health Professions · doi:10.1177/0163278702025001004 · Cited by 198
Estimating Time to Conduct a Meta-analysis From Number of Citations Retrieved
Allen, I. E. (1999) · JAMA: The Journal of the American Medical Association · doi:10.1001/jama.282.7.634 · Cited by 163
Should meta-analysts search Embase in addition to Medline?
Sampson, M. (2003) · Journal of Clinical Epidemiology · doi:10.1016/s0895-4356(03)00110-0 · Cited by 160
Self-citationDoust, J. A., Pietrzak, E., Sanders, S., & Glasziou, P. P. (2005) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2004.09.011 · Cited by 106
Cited by
Searching for and selecting studies
Lefebvre, C., Glanville, J., Briscoe, S., Littlewood, A., Marshall, C., Metzendorf, M., Noel‐Storr, A., Rader, T., Shokraneh, F., Thomas, J., et al. (2019) · Cochrane Handbook for Systematic Reviews of Interventions · doi:10.1002/9781119536604.ch4 · Cited by 647
Boeker, M., Vach, W., & Motschall, E. (2013) · BMC Medical Research Methodology · doi:10.1186/1471-2288-13-131 · Cited by 225
Noel-Storr, A. H., Dooley, G., Wisniewski, S., Glanville, J., Thomas, J., Cox, S., Featherstone, R., & Foxlee, R. (2020) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2020.08.008 · Cited by 136
Retrieving Clinical Evidence: A Comparison of PubMed and Google Scholar for Quick Clinical Searches
Shariff, S. Z., Bejaimal, S. A., Sontrop, J. M., Iansavichus, A. V., Haynes, R. B., Weir, M. A., & Garg, A. X. (2013) · Journal of Medical Internet Research · doi:10.2196/jmir.2624 · Cited by 119
Taljaard, M., McGowan, J., Grimshaw, J. M., Brehaut, J. C., McRae, A., Eccles, M. P., & Donner, A. (2010) · BMC Medical Research Methodology · doi:10.1186/1471-2288-10-15 · Cited by 62
Metadata
- Authors
- Sampson, M., Zhang, L., Morrison, A., Barrowman, N. J., Clifford, T. J., Platt, R. W., Klassen, T. P., & Moher, D.
- Venue
- BMC Medical Research Methodology
- Published
- 2006-07-18
- Volume
- 6
- Issue
- 1
- Pages
- 33-33
- ISSN
- 1471-2288 (ISSN-L)
- OpenAlex
- W1540816673
- Cited by
- 131
- References
- 9
- FWCI
- 4.14
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: An alternative to the hand searching gold standard: validating methodological search filters using relative recall. https://app.cochat.ai/papers/10.1186/1471-2288-6-33
- The paper
- Sampson, M., Zhang, L., Morrison, A., Barrowman, N. J., Clifford, T. J., Platt, R. W., Klassen, T. P., & Moher, D. (2006). An alternative to the hand searching gold standard: validating methodological search filters using relative recall. BMC Medical Research Methodology. https://doi.org/10.1186/1471-2288-6-33
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.