CoChat
Sign Up
Cited by 137Meta-analysis and systematic reviewsscientometrics and bibliometrics researchInformation Retrieval and Search Behavior

A comparison of results of empirical studies of supplementary search techniques and recommendations in review methodology handbooks: a methodological review

Open access
Cooper, Chris, Booth, Andrew, Britten, Nicky, Garside, Ruth·2017·Systematic Reviews
DOI 10.1186/s13643-017-0625-1PMC PMC5704629
Open this paper in CoChat — free accountOpen PDF

Abstract

BACKGROUND: The purpose and contribution of supplementary search methods in systematic reviews is increasingly acknowledged. Numerous studies have demonstrated their potential in identifying studies or study data that would have been missed by bibliographic database searching alone. What is less certain is how supplementary search methods actually work, how they are applied, and the consequent advantages, disadvantages and resource implications of each search method. The aim of this study is to compare current practice in using supplementary search methods with methodological guidance. METHODS: Four methodological handbooks in informing systematic review practice in the UK were read and audited to establish current methodological guidance. Studies evaluating the use of supplementary search methods were identified by searching five bibliographic databases. Studies were included if they (1) reported practical application of a supplementary search method (descriptive) or (2) examined the utility of a supplementary search method (analytical) or (3) identified/explored factors that impact on the utility of a supplementary method, when applied in practice. RESULTS: Thirty-five studies were included in this review in addition to the four methodological handbooks. Studies were published between 1989 and 2016, and dates of publication of the handbooks ranged from 1994 to 2014. Five supplementary search methods were reviewed: contacting study authors, citation chasing, handsearching, searching trial registers and web searching. CONCLUSIONS: There is reasonable consistency between recommended best practice (handbooks) and current practice (methodological studies) as it relates to the application of supplementary search methods. The methodological studies provide useful information on the effectiveness of the supplementary search methods, often seeking to evaluate aspects of the method to improve effectiveness or efficiency. In this way, the studies advance the understanding of the supplementary search methods. Further research is required, however, so that a rational choice can be made about which supplementary search strategies should be used, and when.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1186/s13643-017-0625-1

Cites

10 of 10 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • The methodology is grounded in a comprehensive literature review adhering to rigorous criteria and methodological recommendations established in the scientific literature, specifically the criteria proposed by [58] and the practical guidelines for methodological reviews outlined by [59].

    “The methodology employed in this study is grounded in a comprehensive literature review, adhering to rigorous criteria and methodological recommendations established in the scientific literature.To ensure the validity, clarity, and utility of the analysis, the criteria proposed by [58] and the practical guidelines for methodological reviews outlined by [59] were applied.”

    From Towards Sustainability in Hydraulic Machinery Manufacturing by 3D Printing (2024) · cites this paper as [58]

    The Methods and Study selection sections describe a structured methodological review, including systematic study identification, inclusion criteria and data extraction. They do not set out general practical guidelines for methodological reviews; the procedures described are specific to this paper’s comparison of supplementary search techniques and handbook recommendations. The citation therefore supports an example of review practice, but not the claim that general guidelines from this source were applied. (LLM, conf=0.82)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 49

  1. [2]Hinde S Spackman E Bidirectional citation searching to completion: an exploration of literature searching methods PharmacoEconomics 2015 33 1 5 11 10.1007/s40273-014-0205-3 25145803 · DOI
  2. [3]Papaioannou D Sutton A Carroll C Booth A Wong R Literature searching for social science systematic reviews: consideration of a range of search techniques Health Inf Libr J 2010 27 2 114 122 10.1111/j.1471-1842.2009.00863.x 20565552 · DOI · Paper page
  3. [4]Pearson M Moxham T Ashton K Effectiveness of search strategies for qualitative research about barriers and facilitators of program delivery Eval Health Prof 2011 34 3 297 308 10.1177/0163278710388029 21224267 · DOI
  4. [6]McManus RJ Wilson S Delaney BC Fitzmaurice DA Hyde CJ Tobias RS Jowett S Hobbs FDR Review of the usefulness of contacting other experts when conducting a literature search for systematic reviews BMJ 1998 317 7172 1562 1563 10.1136/bmj.317.7172.1562 9836655 PMC28735 · DOI
  5. [7]Westphal A Kriston L Holzel LP Harter M von Wolff A Efficiency and contribution of strategies for finding randomized controlled trials: a case study from a systematic review on therapeutic interventions of chronic depression J Public Health Res 2014 3 2 177 10.4081/jphr.2014.177 25343133 PMC4207021 · DOI
  6. [8]Kwon Y Powelson SE Wong H Ghali WA Conly JM An assessment of the efficacy of searching in biomedical databases beyond MEDLINE in identifying studies for a systematic review on ward closures as an infection control intervention to control outbreaks Syst Rev 2014 3 135 10.1186/2046-4053-3-135 25387523 PMC4231196 · DOI
  7. [9]Lefebvre C, Manheimer E, Glanville J: Chapter 6: searching for studies. In: Cochrane handbook for systematic reviews of interventions. Edited by Higgins JPT, Green S (editors); 2011. Available from www.handbook.cochrane.org. Accessed July 2016
  8. [12]NICE Guide to the methods of technology appraisal 2013 27905712
  9. [13]Higgins JPT, Deeks JJ (eds.): Chapter 7: Selecting studies and collecting data. in: Higgins JPT, Green S (editors), Cochrane Handbook for Systematic Reviews of Interventions Version 5.1.0 (updated March 2011). The Cochrane Collaboration, 2011. Available from www.handbook.cochrane.org. Accessed July 2016.
  10. [14]Gibson CA Bailey BW Carper MJ LeCheminant JD Kirk EP Huang G DuBose KD Donnelly JE Author contacts for retrieval of data for a meta-analysis on exercise and diet restriction Int J Technol Assess Health Care 2006 22 2 267 270 10.1017/S0266462306051105 16571204 · DOI
  11. [15]O’Leary F Is email a reliable means of contacting authors of previously published papers? A study of the Emergency Medicine Journal for 2001 Emerg Med J 2003 20 4 352 353 10.1136/emj.20.4.352 12835347 PMC1726140 · DOI
  12. [16]Selph SS Ginsburg AD Chou R Impact of contacting study authors to obtain additional data for systematic reviews: diagnostic accuracy studies for hepatic fibrosis Systematic Reviews 2014 3 1 107 10.1186/2046-4053-3-107 25239493 PMC4185334 · DOI
  13. [17]Hetherington J Dickersin K Chalmers I Meinert CL Retrospective and prospective identification of unpublished controlled trials: lessons from a survey of obstetricians and pediatricians Pediatrics 1989 84 2 374 380 2748270
  14. [18]Reveiz L Cardona AF Ospina EG de Agular S An e-mail survey identified unpublished studies for systematic reviews J Clin Epidemiol 2006 59 7 755 758 10.1016/j.jclinepi.2005.11.022 16765280 · DOI
  15. [19]Bakkalbasi N Bauer K Glover J Wang L Three options for citation tracking: Google Scholar, Scopus and Web of Science Biomedical Digital Libraries 2006 3 7 7 10.1186/1742-5581-3-7 16805916 PMC1533854 · DOI · Paper page
  16. [20]Janssens AC Gwinn M Novel citation-based search method for scientific literature: application to meta-analyses BMC Med Res Methodol 2015 15 84 10.1186/s12874-015-0077-z 26462491 PMC4604708 · DOI
  17. [21]Robinson KA Dunn AG Tsafnat G Glasziou P Citation networks of related trials are often disconnected: implications for bidirectional citation searches J Clin Epidemiol 2014 67 7 793 799 10.1016/j.jclinepi.2013.11.015 24725642 · DOI
  18. [25]Adams CE Power A Frederick K Lefebvre C An investigation of the adequacy of MEDLINE searches for randomized controlled trials (RCTs) of the effects of mental health care Psychol Med 1994 24 3 741 748 10.1017/S0033291700027896 7991756 · DOI
  19. [26]Hay PJ Adams CE Lefebvre C The efficiency of searches for randomized controlled trials in the International Journal of Eating Disorders: a comparison of handsearching EMBASE and PsycLIT Health Libraries Review 1996 13 2 91 96 10.1046/j.1365-2532.1996.1320091.x · DOI
  20. [27]Hopewell S Clarke M Lusher A Lefebvre C Westby M A comparison of handsearching versus MEDLINE searching to identify reports of randomized controlled trials Stat Med 2002 21 11 1625 1634 10.1002/sim.1191 12111923 · DOI · Paper page
  21. [28]Jadad AR, McQuay HJ. A high-yield strategy to identify randomized controlled trials for systematic reviews. Online J Curr Clin Trials. 1993;Doc No 33. 8306000
  22. [29]Moher D The OCG: how good are volunteers at searching for published randomized controlled trials? Fundam Clin Pharmacol 1995 9 4 384 386 10.1111/j.1472-8206.1995.tb00514.x 8566939 · DOI
  23. [30]Langham J Thompson E Rowan K Identification of randomized controlled trials from the emergency medicine literature: comparison of hand searching versus MEDLINE searching Ann Emerg Med 1999 34 1 25 34 10.1016/S0196-0644(99)70268-4 10381991 · DOI
  24. [31]Armstrong R Jackson N Doyle J Waters E Howes F It’s in your hands: the value of handsearching in conducting systematic reviews of public health interventions J Public Health 2005 27 4 388 391 10.1093/pubmed/fdi056 16311247 · DOI · Paper page
  25. [32]Blumle A Antes G Handsearching for randomized controlled clinical trials in German medical journals Dtsch Med Wochenschr (1946) 2008 133 6 230 234 10.1055/s-2008-1017501 18236347 · DOI
  26. [33]Croft AM Vassallo DJ Rowe M Handsearching the Journal of the Royal Army Medical Corps for trials J R Army Med Corps 1999 145 2 86 88 10.1136/jramc-145-02-09 10420346 · DOI
  27. [34]Glanville J Cikalo M Crawford F Dozier M McIntosh H Handsearching did not yield additional unique FDG-PET diagnostic test accuracy studies compared with electronic searches: a preliminary investigation Research Synthesis Methods 2012 3 3 202 213 10.1002/jrsm.1046 26062163 · DOI
  28. [35]Mattioli S Farioli A Cooke RMT Baldasseroni A Ruotsalainen J Placidi D Curti S Mancini G Fierro M Campo G Hidden effectiveness? Results of hand-searching Italian language journals for occupational health interventions Occup Environ Med 2012 69 7 522 524 10.1136/oemed-2011-100180 22447642 · DOI
  29. [36]Milne R Thorogood M Hand searching the Journal of Epidemiology and Community Health as part of th Cochrane Collaboration J Epidemiol Community Health 1996 50 2 178 181 10.1136/jech.50.2.178 8762385 PMC1060249 · DOI
  30. [37]Lefebvre C Glanville J Wieland LS Coles B Weightman AL Methodological developments in searching for studies for systematic reviews: past, present and future? Syst Rev 2013 2 78 10.1186/2046-4053-2-78 24066664 PMC4015986 · DOI
  31. [39]van Enst WA Scholten RJ Hooft L Identification of additional trials in prospective trial registers for Cochrane systematic reviews PLoS One 2012 7 8 e42812 10.1371/journal.pone.0042812 22916163 PMC3419740 · DOI
  32. [43]Godin K Stapleton J Kirkpatrick SI Hanning RM Leatherdale ST Applying systematic review search methods to the grey literature: a case study examining guidelines for school-based breakfast programs in Canada Systematic Reviews 2015 4 1 138 10.1186/s13643-015-0125-0 26494010 PMC4619264 · DOI · Paper page
  33. [44]Mahood Q Van Eerd D Irvin E Searching for grey literature for systematic reviews: challenges and benefits Research Synthesis Methods 2014 5 3 221 234 10.1002/jrsm.1106 26052848 · DOI · Paper page
  34. [45]Eysenbach G Tuische J Diepgen TL Evaluation of the usefulness of Internet searches to identify unpublished clinical trials for systematic reviews Med Inform Internet Med 2001 26 3 203 218 10.1080/14639230110075459 11706930 · DOI
  35. [46]McGrath J Davies G Soares K Writing to authors of systematic reviews elicited further data in 17% of cases BMJ 1998 316 7131 631 10.1136/bmj.316.7131.631a 9518939 PMC1112654 · DOI
  36. [47]Mullan RJ Flynn DN Carlberg B Tleyjeh IM Kamath CC LaBella ML Erwin PJ Guyatt GH Montori VM Systematic reviewers commonly contact study authors but do so with limited rigor J Clin Epidemiol 2009 62 2 138 142 10.1016/j.jclinepi.2008.08.002 19013767 · DOI
  37. [48]Ogilvie D Egan M Hamilton V Petticrew M Systematic reviews of health effects of social interventions: 2. Best available evidence: how low should you go? J Epidemiol Community Health 2005 59 10 886 892 10.1136/jech.2005.034199 16166365 PMC1732915 · DOI · Paper page
  38. [49]Sterne JAC, Egger M, Moher D (editors). Chapter 10: Addressing reporting biases. In: Higgins JPT, Green S (editors). Cochrane Handbook for Systematic Reviews of Intervention. Version 5.1.0 (updated March 2011). The Cochrane Collaboration, 2011. Available from www.handbook.cochrane.org. Accessed July 2016.
  39. [50]Kostoff RN The use and misuse of citation analysis in research evaluation Scientometrics 1998 43 1 27 43 10.1007/BF02458392 · DOI
  40. [51]Seglen PO Use of citation analysis and other bibliometric methods in evaluation of the quality of research Tidsskr Nor Laegefore 1989 109 31 3229 3224 2595690
  41. [52]JCF d W Zadpoor AA Dodou D The expansion of Google Scholar versus Web of Science: a longitudinal study Scientometrics 2014 98 2 1547 1565 10.1007/s11192-013-1089-2 · DOI
  42. [54]Kloda LA. Use Google Scholar, Scopus and Web of Science for comprehensive citation tracking. Evid Based Libr Inf Pract. 2007;2(3):4.
  43. [55]Choong MK Tsafnat G Role of citation tracking in updating of systematic reviews AMIA Summits Transl Sci Proc 2014 2014 18 25954571 PMC4419760
  44. [56]Lowe J, Peters J, Shields B. Methods to update systematic literature searches: full update searching vs. forward citation chasing: A case study from a systematic review of diagnostic test accuracy. Poster presented at InterTASC ISSG: 2014; Exeter.
  45. [57]PubMed Commons (Elke Hausner) [ https://www.ncbi.nlm.nih.gov/pubmed/22916163 - comments]. Accessed July 2016.
  46. [58]De Angelis C Drazen JM Frizelle FAP Haug C Hoey J Horton R Kotzin S Laine C Marusic A Overbeke AJPM Clinical trial registration: a statement from the International Committee of Medical Journal Editors N Engl J Med 2004 351 12 1250 1251 10.1056/NEJMe048225 15356289 · DOI · Paper page
  47. [59]Hartung DM Zarin DA Guise JM McDonagh M Paynter R Helfand M Reporting discrepancies between the ClinicalTrials.gov results database and peer-reviewed publications Ann Intern Med 2014 160 7 477 483 10.7326/M13-0480 24687070 PMC4617780 · DOI
  48. [60]Ogino D Takahashi K Sato H Characteristics of clinical trial websites: information distribution between ClinicalTrials.gov and 13 primary registries in the WHO registry network Trials 2014 15 428 10.1186/1745-6215-15-428 25373358 PMC4234832 · DOI
  49. [61]Saleh AA Ratajeski MA Bertolet M Grey literature searching for health sciences systematic reviews: a prospective study of time spent and resources utilized Evidence based library and information practice 2014 9 3 28 50 10.18438/B8DW3K 25914722 PMC4405801 · DOI

Cited by

Sorted by citations

Authors
Cooper, C., Booth, A., Britten, N., & Garside, R.
Affiliations
  • University of Exeter (GB)
  • University of Sheffield (GB)
Corresponding author
Cooper, Chris
ORCID
Venue
Systematic Reviews
Publisher
Springer Science and Business Media LLC
Published
2017-11-28
Volume
6
Issue
1
Pages
234-234
ISSN
2046-4053 (ISSN-L)
DOI
10.1186/s13643-017-0625-1
PMC
PMC5704629
OpenAlex
W2768314176
Cited by
137
References
10
Citations by year
2026: 14 · 2025: 13 · 2024: 18 · 2023: 11 · 2022: 21 · 2021: 20 · 2020: 22 · 2019: 10 · 2018: 8
FWCI
2.71
Type
journal article
Language
en
Open access
Licence
cc-by
Locations
  • Systematic Reviews (publishedVersion, cc-by)
  • Europe PMC (PubMed Central) (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • Open Research Exeter (University of Exeter) (publishedVersion, cc-by)
  • White Rose Research Online (University of Leeds, The University of Sheffield, University of York) (acceptedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
  • UCL Discovery (University College London) (submittedVersion)
  • Figshare (submittedVersion)
  • Explore Bristol Research (submittedVersion, other-oa)
Funders
National Institute for Health Research
Topics
  • Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
  • Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › scientometrics and bibliometrics research
  • Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
Keywords
systematic review methodology, web search, bibliographic database searching
MeSH terms
United Kingdom, Humans, Review Literature as Topic, Databases, Bibliographic, Information Storage and Retrieval, Search Engine

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review