Methodological developments in searching for studies for systematic reviews: past, present and future?
Open accessThe DOI matches its Crossref record; 24 of the 24 references it cites were found in a registry or an index.
Abstract
The Cochrane Collaboration was established in 1993, following the opening of the UK Cochrane Centre in 1992, at a time when searching for studies for inclusion in systematic reviews was not well-developed. Review authors largely conducted their own searches or depended on medical librarians, who often possessed limited awareness and experience of systematic reviews. Guidance on the conduct and reporting of searches was limited. When work began to identify reports of randomized controlled trials (RCTs) for inclusion in Cochrane Reviews in 1992, there were only approximately 20,000 reports indexed as RCTs in MEDLINE and none indexed as RCTs in Embase. No search filters had been developed with the aim of identifying all RCTs in MEDLINE or other major databases. This presented The Cochrane Collaboration with a considerable challenge in identifying relevant studies.Over time, the number of studies indexed as RCTs in the major databases has grown considerably and the Cochrane Central Register of Controlled Trials (CENTRAL) has become the best single source of published controlled trials, with approximately 700,000 records, including records identified by the Collaboration from Embase and MEDLINE. Search filters for various study types, including systematic reviews and the Cochrane Highly Sensitive Search Strategies for RCTs, have been developed. There have been considerable advances in the evidence base for methodological aspects of information retrieval. The Cochrane Handbook for Systematic Reviews of Interventions now provides detailed guidance on the conduct and reporting of searches. Initiatives across The Cochrane Collaboration to improve the quality inter alia of information retrieval include: the recently introduced Methodological Expectations for Cochrane Intervention Reviews (MECIR) programme, which stipulates 'mandatory' and 'highly desirable' standards for various aspects of review conduct and reporting including searching, the development of Standard Training Materials for Cochrane Reviews and work on peer review of electronic search strategies. Almost all Cochrane Review Groups and some Cochrane Centres and Fields now have a Trials Search Co-ordinator responsible for study identification and medical librarians and other information specialists are increasingly experienced in searching for studies for systematic reviews.Prospective registration of clinical trials is increasing and searching trials registers is now mandatory for Cochrane Reviews, where relevant. Portals such as the WHO International Clinical Trials Registry Platform (ICTRP) are likely to become increasingly attractive, given concerns about the number of trials which may not be registered and/or published. The importance of access to information from regulatory and reimbursement agencies is likely to increase. Cross-database searching, gateways or portals and improved access to full-text databases will impact on how searches are conducted and reported, as will services such as Google Scholar, Scopus and Web of Science. Technologies such as textual analysis, semantic analysis, text mining and data linkage will have a major impact on the search process but efficient and effective updating of reviews may remain a challenge.In twenty years' time, we envisage that the impact of universal social networking, as well as national and international legislation, will mean that all trials involving humans will be registered at inception and detailed trial results will be routinely available to all. Challenges will remain, however, to ensure the discoverability of relevant information in diverse and often complex sources and the availability of metadata to provide the most efficient access to information. We envisage an ongoing role for information professionals as experts in identifying new resources, researching efficient ways to link or mine them for relevant data and managing their content for the efficient production of systematic reviews.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Liberati, A., Altman, D. G., Tetzlaff, J., Mulrow, C., Gøtzsche, P. C., Ioannidis, J. P. A., Clarke, M., Devereaux, P. J., Kleijnen, J., & Moher, D. (2009) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2009.06.006 · Cited by 10,578
Development of AMSTAR: A measurement tool to assess the methodological quality of systematic reviews
Shea, B. J., Grimshaw, J. M., Wells, G. A., Boers, M., Andersson, N., Hamel, C., Porter, A. C., Tugwell, P., Moher, D., & Bouter, L. M. (2007) · BMC Medical Research Methodology · doi:10.1186/1471-2288-7-10 · Cited by 4,374
CONSORT 2010 statement: Updated guidelines for reporting parallel group randomised trials
Schulz, K. F., Altman, D. G., & Moher, D. (2010) · Journal of Pharmacology and Pharmacotherapeutics · doi:10.4103/0976-500x.72352 · Cited by 2,165
CONSORT 2010 Statement: updated guidelines for reporting parallel group randomised trials
the CONSORT Group, Schulz, K. F., Altman, D. G., & Moher, D. (2010) · Trials · doi:10.1186/1745-6215-11-32 · Cited by 1,871
Moher, D., Hopewell, S., Schulz, K. F., Montori, V., Gøtzsche, P. C., Devereaux, P. J., Elbourne, D., Egger, M., & Altman, D. G. (2010) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2010.03.004 · Cited by 1,858
meta-analysis bias in location and selection of studies
Egger, M., & Smith, G. D. (1998) · BMJ · doi:10.1136/bmj.316.7124.61 · Cited by 1,374
Clinical Trial Registration: A Statement from the International Committee of Medical Journal Editors
De Angelis, C., Drazen, J. M., Frizelle, F. A., Haug, C., Hoey, J., Horton, R., Kotzin, S., Laine, C., Marusic, A., Overbeke, A. J. P. M., et al. (2004) · New England Journal of Medicine · doi:10.1056/nejme048225 · Cited by 1,241
Full publication of results initially presented in abstracts.
Scherer, R. W., Langenberg, P., & von Elm, E. (2007) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.mr000005.pub3 · Cited by 997
The ClinicalTrials.gov Results Database — Update and Key Issues
Zarin, D. A., Tse, T., Williams, R. J., Califf, R. M., & Ide, N. C. (2011) · New England Journal of Medicine · doi:10.1056/nejmsa1012065 · Cited by 838
Grey literature in meta-analyses of randomized trials of health care interventions
Hopewell, S., McDonald, S., Clarke, M. J., & Egger, M. (2007) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.mr000010.pub3 · Cited by 598 · cited 1×
“where publishing is not the primary activity of the producing body’ [48,49].”
Background
Research works printed but not linked: 66
- Wilczynski NL, Walker CJ, McKibbon KA, Haynes RB: Assessment of methodologic search filters in MEDLINE. Proc Annu Symp Comput Appl Med Care. 1993, 601-605.
- Dickersin K, Scherer R, Lefebvre C: Identifying relevant studies for systematic reviews. BMJ. 1994, 309: 1286-1291. · DOI · Paper page
- PROFILE: Excerpta Medica Newsl. 1994, 11: 2-
- Lefebvre C: Identification of randomized controlled trials using MEDLINE: the situation in 1993. An Evidence-based Health Care System: the Case for Clinical Trials Registries. Edited by: Armstrong C. 1994, Bethesda: National Insitutes of Health, Office of Medical Applications of Research, 23-28.
- Wieland LS, Robinson KA, Dickersin K: Understanding why evidence from randomised clinical trials may not be retrieved from Medline: comparison of indexed and non-indexed records. BMJ. 2012, 344: d7501-doi: 10.1136/bmj.d7501 · DOI · Paper page
- Golder S, Loke Y: The performance of adverse effects search filters in MEDLINE and EMBASE. Health Info Libr J. 2012, 29: 141-151. · DOI
- Golder S, Loke Y: Failure or success of electronic search strategies to identify adverse effects data. J Med Libr Assoc. 2012, 100: 130-134. · DOI
- Golder S, Loke Y, Zorzela L: Some improvements are apparent in identifying adverse effects in systematic reviews from 1994 to 2011. Clin Epidemiol. 2013, 66: 253-260. · DOI
- Golder S, Loke Y, McIntosh HM: Poor reporting and inadequate searches were apparent in systematic reviews of adverse effects. J Clin Epidemiol. 2008, 61: 440-448. · DOI
- Beynon R, Leeflang MM, McDonald S, Eisinga A, Mitchell RL, Whiting P, Glanville JM: Search strategies to identify diagnostic accuracy studies in MEDLINE and EMBASE. Cochrane Database Syst Rev. 2013, 9: MR000022-
- Bachmann LM, Estermann P, Kronenberg C, Ter Riet G: Identifying diagnostic accuracy studies in EMBASE. J Med Libr Assoc. 2003, 91: 341-346.
- Ritchie G, Glanville J, Lefebvre C: Do published search filters to identify diagnostic test accuracy studies perform adequately?. Health Info Libr J. 2007, 24: 188-192. · DOI · Paper page
- Glanville J, Wilson P, Richardson R: Accessing the online evidence: a guide to key sources of research information on clinical and cost effectiveness. Qual Saf Health Care. 2003, 12: 229-231. · doi:10.1136/qhc.12.3.229
- Glanville J, Kaunelis D, Mensinkai S: How well do search filters perform in identifying economic evaluations in MEDLINE and EMBASE?. Int J Technol Assess Health Care. 2009, 25: 522-529. · DOI · Paper page
- Glanville J, Paisley S: Identifying economic evaluations for health technology assessment. Int J Technol Assess Health Care. 2010, 26: 436-440. · DOI
- McKinlay RJ, Wilczynski NL, Haynes RB: Optimal search strategies for detecting cost and economic studies in EMBASE. BMC Health Serv Res. 2006, 6: 67- · DOI
- Wilczynski NL, Haynes RB: Developing optimal search strategies for detecting clinically sound prognostic studies in MEDLINE: an analytic survey. BMC Med. 2004, 2: 23- · DOI · Paper page
- Wilczynski NL, Haynes RB: Optimal search strategies for detecting clinically sound prognostic studies in EMBASE: an analytic survey. J Am Med Inform Assoc. 2005, 12: 481-485. · DOI · Paper page
- Walker-Dilks C, Wilczynski NL, Haynes RB: Cumulative index to nursing and allied health literature search strategies for identifying methodologically sound causation and prognosis studies. Appl Nurs Res. 2008, 21: 98-103. · DOI
- Wilczynski NL, Haynes RB: Developing optimal search strategies for detecting clinically sound causation studies in MEDLINE. AMIA Annu Symp Proc. 2003, 719-723.
- Haynes RB, Kastner M, Wilczynski NL: Developing optimal search strategies for detecting clinically sound and relevant causation studies in EMBASE. BMC Med Inform Decis Mak. 2005, 5: 8- · DOI · Paper page
- Wieland S, Dickersin K: Selective exposure reporting and Medline indexing limited the search sensitivity for observational studies of the adverse effects of oral contraceptives. J Clin Epidemiol. 2005, 58: 560-567. · DOI
- Grant MJ: How does your searching grow? A survey of search preferences and the use of optimal search strategies in the identification of qualitative research. Health Info Libr J. 2004, 21: 21-32. · DOI
- Wong SS, Wilczynski NL, Haynes RB: Developing optimal search strategies for detecting clinically relevant qualitative studies in MEDLINE. Stud Health Technol Inform. 2004, 107: 311-316.
- McKibbon KA, Wilczynski NL, Haynes RB: Developing optimal search strategies for retrieving qualitative studies in PsycINFO. Eval Health Prof. 2006, 29: 440-454. · DOI · Paper page
- Flemming K, Briggs M: Electronic searching to locate qualitative research: evaluation of three strategies. J Adv Nurs. 2007, 57: 95-100. · DOI · Paper page
- Boynton J, Glanville J, McDaid D, Lefebvre C: Identifying systematic reviews in MEDLINE: developing an objective approach to search strategy design. J Info Sci. 1998, 24: 137-157. · DOI · Paper page
- White V, Glanville J, Lefebvre C, Sheldon T: A statistical approach to designing search filters to find systematic reviews: objectivity enhances accuracy. J Info Sci. 2001, 27: 357-370. · DOI · Paper page
- Wong SS, Wilczynski NL, Haynes RB: Comparison of top-performing search strategies for detecting clinically sound treatment studies and systematic reviews in MEDLINE and EMBASE. J Med Libr Assoc. 2006, 94: 451-455.
- Wilczynski NL, Haynes RB: EMBASE search strategies achieved high sensitivity and specificity for retrieving methodologically sound systematic reviews. J Clin Epidemiol. 2007, 60: 29-33. · DOI
- Glanville JM, Lefebvre C, Miles JN, Camosso-Stefinovic J: How to identify randomized controlled trials in MEDLINE: ten years on. J Med Libr Assoc. 2006, 94: 130-136.
- Lefebvre C, Manheimer E, Glanville J: Chapter 6: searching for studies. In Cochrane Handbook for Systematic Reviews of Interventions. Version 5.1.0. Edited by Higgins JPT, Green S. The Cochrane Collaboration; 2011 http://www.handbook.cochrane.org
- Glanville J, Bayliss S, Booth A, Dundar Y, Fernandes H, Fleeman ND, Foster L, Fraser C, Fry-Smith A, Golder S, Lefebvre C, Miller C, Paisley S, Payne L, Price A, Welch K: So many filters, so little time: the development of a search filter appraisal checklist. J Med Libr Assoc. 2008, 96: 356-361. · DOI · Paper page
- Sampson M, McGowan J, Lefebvre C, Moher D, Grimshaw J: PRESS: Peer Review of Electronic Search Strategies [technology report number 477]. 2008, Ottawa: Canadian Agency for Drugs and Technologies in Health, http://www.cadth.ca/media/pdf/477_PRESS-Peer-Review-Electronic-Search-Strategies_tr_e.pdf ,
- Sampson M, McGowan J, Cogo E, Grimshaw J, Moher D, Lefebvre C: An evidence-based practice guideline for the peer review of electronic search strategies. J Clin Epidemiol. 2009, 62: 944-952. · DOI · Paper page
- Hopewell S, Clarke M, Lefebvre C, Scherer R: Handsearching versus electronic searching to identify reports of randomized trials. Cochrane Database Syst Rev. 2007, 2: MR000001-
- McDonald S, Lefebvre C, Antes G, Galandi D, Gotzsche P, Hammarquist C, Haugh M, Jensen KL, Kleijnen J, Loep M, Pistotti V, Rüther A: The contribution of handsearching European general health care journals to the Cochrane Controlled Trials Register. Eval Health Prof. 2002, 25: 65-75. · DOI · Paper page
- Horsley T, Dingwall O, Sampson M: Checking reference lists to find additional studies for systematic reviews. Cochrane Database Syst Rev. 2011, 8: MR000026-
- Schlosser RW, Wendt O, Bhavnani S, Nail-Chiwetalu B: Use of information-seeking strategies for developing systematic reviews and engaging in evidence-based practice: the application of traditional and comprehensive pearl growing. A review. Int J Lang Commun Disord. 2006, 41: 567-582. · DOI · Paper page
- Ormstad SS, Isojarvi J: Keeping up to date with information retrieval research: summarized research in information retrieval (SuRe info). Journal of EAHIL. 2013, 9: 17-19.
- MECIR: Methodological Standards for the Conduct of New Cochrane Intervention Reviews. 2012, http://www.editorial-unit.cochrane.org/sites/editorial-unit.cochrane.org/files/uploads/MECIR_conduct_standards%202.2%2017122012.pdf ,
- House of Commons: Science and technology committee: Clinical Trials: Written Evidence. 2013, http://www.parliament.uk/documents/commons-committees/science-technology/Clinical%20trials%20combined.pdf ,
- Goldacre B: AllTrials. 2013, http://www.alltrials.net ,
- Bero L:Searching for unpublished trials using trials registers and trials web sites and obtaining unpublished trial data and corresponding trial protocols from regulatory agencies. 2013, http://methods.cochrane.org/projects-developments/searching-unpublished-trials-using-trials-registers-and-trials-web-sites-and-o ,
- Ghersi D, Pang T: From Mexico to Mali: four years in the history of clinical trial registration. J Evid Based Med. 2009, 2: 1-7. · DOI · Paper page
- Doshi P, Jefferson T, Del Mar C: The imperative to share clinical study reports: recommendations from the Tamiflu experience. PLoS Med. 2012, 9: e1001201- · DOI · Paper page
- Jefferson T, Jones MA, Doshi P, Del Mar CB, Heneghan CJ, Hama R, Thompson MJ: Neuraminidase inhibitors for preventing and treating influenza in healthy adults and children. Cochrane Database Syst Rev. 2012, 1: CD008965-
- Doshi P: Transparency interrupted: the curtailment of the European Medicines Agency’s policy on access to documents. JAMA Intern Med. 2013, doi:10.1001/jamainternmed.2013.9989 · doi:10.1001/jamainternmed.2013.9989
- Doshi P, Jones M, Jefferson T: Rethinking credible evidence synthesis. BMJ. 2012, 344: d7898- · DOI · Paper page
- Jefferson T, Doshi P, Thompson M, Heneghan C: Ensuring safe and effective drugs: who can do what it takes?. BMJ. 2011, 342: c7258- · DOI · Paper page
- Fitzpatrick RB: Science.gov: gateway to government science information. Med Ref Serv Q. 2010, 29: 58-66. · doi:10.1080/02763860903485191
- De Vet H, Eisinga A, Riphagen I, Aertgeerts B, Pewsner D: Chapter 7: searching for studies. In Cochrane Handbook for Systematic Reviews of Diagnostic Test Accuracy version 04, The Cochrane Collaboration. Edited by Deeks J, Bossuyt P, Gatsonis C; 2008. http://srdta.cochrane.org/sites/srdta.cochrane.org/files/uploads/Chapter07-Searching-%28September-2008%29.pdf
- Hausner E, Waffenschmidt S, Kaiser T, Simon M: Routine development of objectively derived search strategies. Syst Rev. 2012, 1: 19- · DOI · Paper page
- Spoor P, Airey M, Bennett C, Greensill J, Williams R: Use of the capture-recapture technique to evaluate the completeness of systematic literature searches. BMJ. 1996, 313: 342-343. · DOI · Paper page
- Rucker G, Reiser V, Motschall E, Binder H, Meerpohl JJ, Antes G, Schumacher M: Boosting qualifies capture-recapture methods for estimating the comprehensiveness of literature searches for systematic reviews. J Clin Epidemiol. 2011, 64: 1364-1372. · DOI
- Sampson M, Zhang L, Morrison A, Barrowman NJ, Clifford TJ, Platt RW, Klassen TP, Moher D: An alternative to the hand searching gold standard: validating methodological search filters using relative recall. BMC Med Res Methodol. 2006, 6: 33- · DOI · Paper page
- Kastner M, Straus S, Goldsmith CH: Estimating the horizon of articles to decide when to stop searching in systematic reviews: an example using a systematic review of RCTs evaluating osteoporosis clinical decision support tools. AMIA Annu Symp Proc. 2007, 389-393.
- Booth A: How much searching is enough? Comprehensive versus optimal retrieval for technology assessments. Int J Technol Assess Health Care. 2010, 26: 431-435. · DOI · Paper page
- Sampson M, Shojania KG, McGowan J, Daniel R, Rader T, Iansavichene AE, Ji J, Ansari MT, Moher D: Surveillance search techniques identified the need to update systematic reviews. J Clin Epidemiol. 2008, 61: 755-762. · DOI
- Shekelle PG, Newberry SJ, Wu H, Suttorp M, Motala A, Lim YW, Balk EM, Chung M, Yu WW, Lee J, Gaylor JM, Moher D, Ansari MT, Skidmore R, Garritty C: Identifying Signals for Updating Systematic Reviews: A Comparison of Two Methods [internet]. 2011, Rockville, MD: Agency for Healthcare Research and Quality (US), http://www.ncbi.nlm.nih.gov/books/NBK56774/ ,
- CRD: Systematic Reviews: CRD’s Guidance for Undertaking Reviews in Health Care. 2009, York: Centre for Reviews and Dissemination, University of York
- IOM: Standards for finding and assessing individual studies. Finding What Works in Health Care: Standards for Systematic Reviews, Committee on Standards for Systematic Reviews of Comparative Effectiveness Research. Edited by: Eden J, Levit L, Berg A, Morton S. 2011, Washington, DC: The National Academies Press, 81-154.
- Moher D, Liberati A, Tetzlaff J, Altman DG: Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. J Clin Epidemiol. 2009, 62: 1006-1012. · DOI · Paper page
- Ellis AK: Teaching and Learning Elementary Social Studies. 1970, Boston, MA: Allyn & Bacon
- Cepeda MS, Lobanov V, Berlin JA: From ClinicalTrials.gov trial registry to an analysis-ready database of clinical trial results. Clin Trials. 2013, 10: 347-348. · DOI
- Ip S, Hadar N, Keefe S, Parkin C, Iovin R, Balk EM, Lau J: A web-based archive of systematic review data. Syst Rev. 2012, 1: 15- · DOI · Paper page
Cited by
Searching for and selecting studies
Lefebvre, C., Glanville, J., Briscoe, S., Littlewood, A., Marshall, C., Metzendorf, M., Noel‐Storr, A., Rader, T., Shokraneh, F., Thomas, J., et al. (2019) · Cochrane Handbook for Systematic Reviews of Interventions · doi:10.1002/9781119536604.ch4 · Cited by 647
Self-citationBooth, A. (2016) · Systematic Reviews · doi:10.1186/s13643-016-0249-x · Cited by 569
Mendes, K. D. S., Silveira, R. C. D. C. P., & Galvão, C. M. (2019) · Texto & Contexto - Enfermagem · doi:10.1590/1980-265x-tce-2017-0204 · Cited by 548
Adams, J., Hillier-Brown, F. C., Moore, H. J., Lake, A. A., Araujo-Soares, V., White, M., & Summerbell, C. (2016) · Systematic Reviews · doi:10.1186/s13643-016-0337-y · Cited by 486
“In addition, developing better searching and retrieval methods should also facilitate syntheses of grey literature and information, particularly using more sophisticated methods for internet searching such as text analytics or data mining [7, 52].”
MethodRethlefsen, M. L., Farrell, A. M., Osterhaus Trzasko, L. C., & Brigham, T. J. (2015) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2014.11.025 · Cited by 331
Metadata
- Authors
- Lefebvre, C., Glanville, J., Wieland, L. S., Coles, B., & Weightman, A. L.
- Venue
- Systematic Reviews
- Published
- 2013-09-25
- Volume
- 2
- Issue
- 1
- Pages
- 78-78
- ISSN
- 2046-4053 (ISSN-L)
- OpenAlex
- W2130035612
- Cited by
- 112
- References
- 24
- FWCI
- 2.32
- Type
- journal article
- Language
- en
- Open access
- gold
- Licence
- creativecommons.org/licenses/by/2.0 (tdm)
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Methodological developments in searching for studies for systematic reviews: past, present and future? https://app.cochat.ai/papers/10.1186/2046-4053-2-78
- The paper
- Lefebvre, C., Glanville, J., Wieland, L. S., Coles, B., & Weightman, A. L. (2013). Methodological developments in searching for studies for systematic reviews: past, present and future? Systematic Reviews. https://doi.org/10.1186/2046-4053-2-78
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.