Searching for studies: A guide to information retrieval for Campbell systematic reviews
Open accessThe DOI matches its Crossref record; 38 of the 38 references it cites were found in a registry or an index.
Abstract
This guide outlines general issues in searching for studies; describes the main sources of potential studies; and discusses how to plan the search process, design, and carry out search strategies, manage references found during the search process and document and report the search process.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
PRISMA-P Group, Moher, D., Shamseer, L., Clarke, M., Ghersi, D., Liberati, A., Petticrew, M., Shekelle, P., & Stewart, L. A. (2015) · Systematic Reviews · doi:10.1186/2046-4053-4-1 · Cited by 26,223 · cited 1×
“P (Moher et al., 2015) extension for reporting systematic review protocols.”
BackgroundPage, M. J., Moher, D., Bossuyt, P. M., Boutron, I., Hoffmann, T. C., Mulrow, C. D., Shamseer, L., Tetzlaff, J. M., Akl, E. A., Brennan, S. E., et al. (2021) · BMJ · doi:10.1136/bmj.n160 · Cited by 13,262
Cochrane Handbook for Systematic Reviews of Interventions
Higgins, J. P. T., Green, S., & Ben Van Den, A. (2020) · International Coaching Psychology Review · doi:10.53841/bpsicpr.2020.15.2.123 · Cited by 5,670
The Handbook of Research Synthesis and Meta-Analysis
(2019) · doi:10.7758/9781610448864 · Cited by 4,553
Rethlefsen, M. L., Kirtley, S., Waffenschmidt, S., Ayala, A. P., Moher, D., Page, M. J., Koffel, J. B., PRISMA-S Group, Blunt, H., Brigham, T., et al. (2021) · Systematic Reviews · doi:10.1186/s13643-020-01542-z · Cited by 3,714 · cited 3×
“S) Extension states that authors ‘ should strongly consider having the search strategy peer reviewed by an experienced searcher, informational specialist, or librarian ’ (Rethlefsen et al., 2021 Page, McKenzie, et al., 2021; Page, Moher, et al., 2021; Rethlefsen et al., 2021).”
BackgroundMethodSelf-citationRethlefsen, M. L., Kirtley, S., Waffenschmidt, S., Ayala, A. P., Moher, D., Page, M. J., & Koffel, J. (2019) · doi:10.31219/osf.io/sfc38 · Cited by 3,696
Systematic Reviews in the Social Sciences: A Practical Guide
Petticrew, M., & Roberts, H. (2006) · doi:10.1002/9780470754887 · Cited by 3,586
JBI Manual for Evidence Synthesis
(2024) · doi:10.46658/jbimes-24-01 · Cited by 2,483
The role of google scholar in evidence reviews and its applicability to grey literature searching
Haddaway, N. R., Collins, A. M., Coughlin, D., & Kirk, S. (2015) · PLOS ONE · doi:10.1371/journal.pone.0138237 · Cited by 2,387
Greenhalgh, T., & Peacock, R. (2005) · BMJ · doi:10.1136/bmj.38636.593461.68 · Cited by 2,045 · cited 3×
“Along with searching the references cited in existing systematic reviews and meta ‐ analyses, reference lists of identified studies may also be searched for additional studies (Greenhalgh & Peacock, 2005; Horsley et al., 2011).”
BackgroundMethodFull publication of results initially presented in abstracts.
Scherer, R. W., Langenberg, P., & von Elm, E. (2007) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.mr000005.pub3 · Cited by 997
Belur, J., Tompson, L., Thornton, A., & Simon, M. (2021) · Sociological Methods & Research · doi:10.1177/0049124118799372 · Cited by 753 · cited 1×
“(Aromataris et al., 2024; McHugh, 2012) to establish a transparent metric for concluding the pilot (Belur et al., 2021).”
BackgroundRoyle, P., & Milne, R. (2003) · International Journal of Technology Assessment in Health Care · doi:10.1017/s0266462303000552 · Cited by 693 · cited 1×
“…about completed but never published studies has not always been found to be fruitful (Hetherington et al., 1989; Horton, 1997) though some researchers have reported that this is an important method for retrieving studies for systematic reviews (Greenhalgh & Peacock, 2005; Royle & Milne, 2003).”
MethodO’Mara-Eves, A., Thomas, J., McNaught, J., Miwa, M., & Ananiadou, S. (2015) · Systematic Reviews · doi:10.1186/2046-4053-4-5 · Cited by 681
Grey literature in meta-analyses of randomized trials of health care interventions
Hopewell, S., McDonald, S., Clarke, M. J., & Egger, M. (2007) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.mr000010.pub3 · Cited by 598 · cited 5×
“In a Cochrane methodology review, all five studies reviewed showed that published trials showed an overall greater treatment effect than grey literature trials (Hopewell et al., 2007).”
BackgroundMethodInfluentialSystematic Reviews: Constructing a Search Strategy and Searching for Evidence
Aromataris, E., & Riitano, D. (2014) · AJN, American Journal of Nursing · doi:10.1097/01.naj.0000446779.99522.f6 · Cited by 523
Searching for studies: a guide to information retrieval for Campbell systematic reviews
Kugley, S., Wade, A., Thomas, J., Mahood, Q., Jørgensen, A. K., Hammerstrøm, K., & Sathe, N. (2017) · Campbell Systematic Reviews · doi:10.4073/cmg.2016.1 · Cited by 371
Rethlefsen, M. L., Farrell, A. M., Osterhaus Trzasko, L. C., & Brigham, T. J. (2015) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2014.11.025 · Cited by 331
Papaioannou, D., Sutton, A., Carroll, C., Booth, A., & Wong, R. (2010) · Health Information & Libraries Journal · doi:10.1111/j.1471-1842.2009.00863.x · Cited by 318
Full publication of results initially presented in abstracts
Scherer, R. W., Meerpohl, J. J., Pfeifer, N., Schmucker, C., Schwarzer, G., & von Elm, E. (2018) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.mr000005.pub4 · Cited by 307
Finding What Works in Health Care: Standards for Systematic Reviews
Committee on Standards for Systematic Reviews of Comparative Effectiveness Research, Board on Health Care Services, & Institute of Medicine (2011) · doi:10.17226/13059 · Cited by 289
Reviewing retrieved references for inclusion in systematic reviews using EndNote
Bramer, W. M., Milic, J., & Mast, F. (2017) · Journal of the Medical Library Association · doi:10.5195/jmla.2017.111 · Cited by 264 · cited 1×
“For review teams that do not have access to screening software or services, reference managers may also be used to review, screen, and select studies (Bramer et al., 2017; Mendes et al., 2019).”
MethodMcKeown, S., & Mir, Z. M. (2021) · Systematic Reviews · doi:10.1186/s13643-021-01583-y · Cited by 251
Alshami, A., Elsayed, M., Ali, E., Eltoukhy, A. E. E., & Zayed, T. (2023) · Systems · doi:10.3390/systems11070351 · Cited by 202
Errors in search strategies used in systematic reviews and their effects on information retrieval
Salvador-Oliván, J. A., Marco-Cuenca, G., & Arquero-Avilés, R. (2019) · Journal of the Medical Library Association · doi:10.5195/jmla.2019.567 · Cited by 199
Royle, P., & Waugh, N. (2003) · Health Technology Assessment · doi:10.3310/hta7340 · Cited by 194 · cited 1×
“…reviews, authors may seek guidance on searching from Chapter 7 in Evidence ‐ Based Decisions and Economics (Glanville & Paisley, 2010) or methodologic reports (Alton et al., 2006; Glanville & Paisley, 2010; Glanville et al., 2009; McKinlay et al., 2006; Royle & Waugh, 2003; Sassi et al., 2002).”
BackgroundErrors in search strategies were identified by type and frequency
Sampson, M., & McGowan, J. (2006) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2006.01.007 · Cited by 142 · cited 1×
“Studies have shown that errors occur in the search strategies underpinning systematic reviews and that search strategies are not always conducted or reported to a high standard (Layton, 2017; Mullins et al., 2014; Ramirez et al., 2022; Salvador ‐ Olivan et al., 2019; Sampson & McGowan, 2006).”
BackgroundBramer, W. M., Giustini, D., Kramer, B. M., & Anderson, P. (2013) · Systematic Reviews · doi:10.1186/2046-4053-2-115 · Cited by 139
Stansfield, C., Dickson, K., & Bangpan, M. (2016) · Systematic Reviews · doi:10.1186/s13643-016-0371-9 · Cited by 122
Self-citationBorissov, N., Haas, Q., Minder, B., Kopp-Heim, D., von Gernler, M., Janka, H., Teodoro, D., & Amini, P. (2022) · Systematic Reviews · doi:10.1186/s13643-022-02045-9 · Cited by 116 · cited 1×
“Review teams may turn to published studies for guidance on best practices for removing duplicates such as considering free tools (Guimarães et al., 2022), comparing reference managers with screening tools (McKeown & Mir, 2021) or turning to newer de ‐ duplication algorithms (Borissov et al., 2022).”
BackgroundMeert, D., Torabi, N., & Costella, J. (2016) · Journal of the Medical Library Association : JMLA · doi:10.3163/1536-5050.104.4.004 · Cited by 108
the COVID-19 NMA Consortium, Oikonomidi, T., Boutron, I., Pierre, O., Cabanac, G., & Ravaud, P. (2020) · BMC Medicine · doi:10.1186/s12916-020-01880-8 · Cited by 54 · cited 1×
“Letters, comments, errata and preprints may all contain relevant information about a study (Iansavichene et al., 2008; Oikonomidi et al., 2020; Zeraatkar et al., 2022).”
BackgroundKastner, M., Straus, S. E., McKibbon, K. A., & Goldsmith, C. H. (2009) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2008.06.001 · Cited by 47 · cited 1×
“Statistical techniques, capture ‐ mark ‐ recapture (Kastner et al., 2009; Lane et al., 2013), or the relative recall technique (Sampson et al., 2006; Sampson & McGowan, 2011), have also been developed to assess performance.”
MethodSystematic review toolbox: a catalogue of tools to support systematic reviews
Marshall, C., & Brereton, P. (2015) · Proceedings of the 19th International Conference on Evaluation and Assessment in Software Engineering · doi:10.1145/2745802.2745824 · Cited by 44
Mullins, M. M., DeLuca, J. B., Crepaz, N., & Lyles, C. M. (2014) · Research Synthesis Methods · doi:10.1002/jrsm.1098 · Cited by 39
Layton, D. (2017) · The International Journal of Prosthodontics · doi:10.11607/ijp.5193 · Cited by 31
Coleman, S., Wright, J. M., Nixon, J., Schoonhoven, L., Twiddy, M., & Greenhalgh, J. (2020) · BMC Medical Research Methodology · doi:10.1186/s12874-020-01084-x · Cited by 20
Should Systematic Reviewers Search for Randomized, Controlled Trials Published as Letters?
Iansavichene, A. E., Sampson, M., McGowan, J., & Ajiferuke, I. S. Y. (2008) · Annals of Internal Medicine · doi:10.7326/0003-4819-148-9-200805060-00023 · Cited by 9 · cited 1×
“Letters, comments, errata and preprints may all contain relevant information about a study (Iansavichene et al., 2008; Oikonomidi et al., 2020; Zeraatkar et al., 2022).”
Background
Research works printed but not linked: 87
- 10.18438/eblip29601 · DOI
- 10.1017/S0266462306051452 · DOI
- 10.5195/jmla.2021.1219 · DOI
- 10.1016/S0749-3797(00)00135-5 · DOI
- 10.1002/asi.23605 · DOI
- 10.1136/ip.2008.018556 · DOI
- 10.1136/bmjgh-2023-013111 · DOI
- 10.1001/archpediatrics.2007.40 · DOI · Paper page
- 10.1186/s13643-016-0249-x · DOI · Paper page
- 10.1016/j.jclinepi.2022.09.004 · DOI
- 10.5596/c10-022 · DOI
- Brietzke E., Gomes F. A., Gerchman F., Freire R. (2023). Should systematic reviews and meta-analysis include data from preprints? Trends in Psychiatry and Psychotherapy, 45, 20210324. https://doi.org/10.47626/2237-6089-2021-0324 · DOI
- 10.1002/jrsm.1275 · DOI · Paper page
- 10.1111/hir.12313 · DOI
- Centre for Reviews and Dissemination. (2009). Systematic Reviews: CRD's guidance for undertaking reviews in health care. University of York.
- 10.3310/hta7230 · DOI · Paper page
- Clarivate. (2024). Resources for librarians: Web of science coverage details. Clarivate. https://clarivate.libguides.com/librarianresources/coverage
- 10.1186/s12874-018-0545-3 · DOI · Paper page
- 10.1002/jrsm.1485 · DOI
- 10.1016/j.jclinepi.2018.02.025 · DOI
- 10.1007/s12079-016-0370-6 · DOI · Paper page
- 10.11645/13.2.2621 · DOI
- 10.18438/B8KP66 · DOI · Paper page
- Elliot M. Fairweather I. Olsen W. Pampaka M. (Eds.). (n.d.). Text mining (text analytics). In A dictionary of social research methods. Oxford University Press. https://www.oxfordreference.com/view/10.1093/acref/9780191816826.001.0001/acref-9780191816826 · doi:10.1093/acref/9780191816826.001.0001/acref-9780191816826
- 10.1177/0192623320920209 · DOI · Paper page
- EPOC. (2017). Cochrane database syntax guide. Cochrane Effective Practice and Organisation of Care (EPOC). https://epoc.cochrane.org/sites/epoc.cochrane.org/files/uploads/Resources-for-authors2017/database_syntax_guide.pdf
- Eunethta. (2019). Process of information retrieval for systematic reviews and health technology assessments on clinical effectiveness. Methodological Guidelines. EUnetHTA. https://www.eunethta.eu/
- 10.5195/jmla.2020.492 · DOI
- Foster M. J. (2015). An overview of the role of librarians in systematic reviews: From expert search to project manager. Journal of EAHIL, 11, 3–7. http://eahil.eu/wp-content/uploads/2015/09/3-7-Foster.pdf
- 10.1016/j.jclinepi.2023.04.017 · DOI
- 10.1002/jrsm.1528 · DOI
- 10.1111/wvn.12611 · DOI
- 10.3163/1536-5050.96.4.011 · DOI · Paper page
- 10.1017/S0266462309990523 · DOI · Paper page
- 10.1002/9781444320398.ch7 · DOI
- 10.1007/s00423-017-1646-x · DOI · Paper page
- 10.1016/j.jclinepi.2022.10.009 · DOI
- 10.1542/peds.84.2.374 · DOI · Paper page
- 10.1002/jrsm.1635 · DOI
- 10.1136/bmj-2023-078384 · DOI
- 10.1016/S0301-2115(02)00265-8 · DOI
- 10.1002/14651858.MR000026.pub2 · DOI · Paper page
- 10.1016/S0140-6736(05)62564-0 · DOI
- Institute for Quality and Efficiency in Health Care. (2022). IQWiG general methods (Version 6.1). Institute for Quality and Efficiency in Health Care. https://www.iqwig.de/methoden/general-methods_version-6-1.pdf
- 10.1186/2046-4053-3-135 · DOI · Paper page
- 10.1016/j.jcrc.2012.10.071 · DOI
- 10.1017/S0266462321000210 · DOI
- Lefebvre C., Glanville J., Briscoe S., Featherstone R., Littlewood A., Marshall C., Metzendorf M.-I., Noel-Stoff A., Paynter R., Rader T., Thomas J., Wieland L. (2023). Chapter 4: Searching for and selecting studies. In Higgins J., Thomas J., Chandler J., Cumpston M., Li T., Page M., Welch V. (Eds.), Cochrane handbook for systematic reviews of interventions. Cochrane. https://www.training.cochrane.org/handbook
- Littell J. H. (2005). Lessons from a systematic review of effects of multisystemic therapy. Decision Making in Child Welfare, 27(4), 445–463. https://doi.org/10.1016/j.childyouth.2004.11.009 · DOI
- 10.1080/19439342.2012.711342 · DOI · Paper page
- Mallett S. Hopewell S. Clarke M. (2002). Grey literature in systematic reviews: The first 1000 Cochrane systematic reviews. Fourth Symposium on Systematic Reviews: Pushing the Boundaries Oxford.
- 10.1136/bmj.325.7363.528 · DOI · Paper page
- 10.11613/BM.2012.031 · DOI · Paper page
- 10.1186/1472-6963-6-67 · DOI
- 10.1097/BSD.0000000000001532 · doi:10.1097/bsd.0000000000001532
- 10.1590/1980-265x-tce-2017-0204 · DOI · Paper page
- Metzendorf M. I. (2016). Why medical information specialists should routinely form part of teams producing high quality systematic reviews – A Cochrane perspective. Journal of the European Association for Health Information and Libraries, 12(4), 6–9.
- 10.1038/549023a · DOI · Paper page
- 10.11124/JBIES-21-00138 · DOI · Paper page
- 10.1111/hir.12367 · DOI
- 10.1136/jech.2005.034181 · DOI · Paper page
- 10.1016/j.jclinepi.2021.03.001 · DOI · Paper page
- 10.1016/j.jclinepi.2023.111237 · DOI
- Pullin A. S., Frampton G. K., Livoreil B., Petrokofsky G. (Eds.). (2022). Guidelines and standards for evidence synthesis in environmental management: Vol. 5.1. Collaboration for Environmental Evidence. www.environmentalevidence.org/information-for-authors
- 10.1016/j.acalib.2022.102567 · DOI
- 10.1186/s13643-016-0197-5 · DOI · Paper page
- 10.1186/s13643-021-01733-2 · DOI
- 10.1038/sj.ebd.6400602 · DOI
- 10.1002/cl2.1380 · DOI
- 10.5195/jmla.2019.491 · DOI
- 10.1002/jrsm.40 · DOI · Paper page
- 10.1186/1471-2288-6-33 · DOI · Paper page
- 10.1097/00005650-200205000-00004 · DOI · Paper page
- 10.1007/s13312-022-2478-5 · DOI
- 10.1371/journal.pone.0256833 · DOI
- Scopus. (2024). Scopus content. Elsevier. https://www.elsevier.com/products/scopus/content
- 10.1080/19439342.2018.1438495 · DOI
- 10.5195/jmla.2018.82 · DOI · Paper page
- Wang S.-S., Lin W.-Y. C. (2022). Medical librarians participating in systematic reviews: Perspectives of citation analysis. Journal of Educational Media & Library Sciences, 59(1), 5–34.
- 10.1080/13645579.2021.1892378 · DOI · Paper page
- White H. (2009). Scientific communication and literature retrieval. In Cooper H., Hedges L., Valentine J. (Eds.), The handbook of research synthesis (2nd ed.). Russell Sage Foundation.
- 10.1111/j.1471-1842.2008.00823.x · DOI · Paper page
- 10.1016/j.jclinepi.2006.04.001 · DOI
- 10.1111/j.1547-5069.2006.00100.x · DOI · Paper page
- 10.3163/1536-5050.98.2.006 · DOI
- 10.1016/j.cpr.2012.06.007 · DOI
- 10.1136/bmjmed-2022-000309 · DOI · Paper page
Cited by
Rethlefsen, M. L., Kirtley, S., Waffenschmidt, S., Ayala, A. P., Moher, D., Page, M. J., & Koffel, J. (2019) · doi:10.31219/osf.io/sfc38 · Cited by 3,696
Rethlefsen, M. L., Kirtley, S., Waffenschmidt, S., Ayala, A. P., Moher, D., Page, M. J., Koffel, J. B., PRISMA-S Group, Blunt, H., Brigham, T., et al. (2021) · Systematic Reviews · doi:10.1186/s13643-020-01542-z · Cited by 3,714
Self-citationA systematic approach to searching: an efficient and complete method to develop literature searches
Bramer, W. M., De Jonge, G. B., Rethlefsen, M. L., Mast, F., & Kleijnen, J. (2018) · Journal of the Medical Library Association · doi:10.5195/jmla.2018.283 · Cited by 863
O’Mara-Eves, A., Thomas, J., McNaught, J., Miwa, M., & Ananiadou, S. (2015) · Systematic Reviews · doi:10.1186/2046-4053-4-5 · Cited by 681
Adams, J., Hillier-Brown, F. C., Moore, H. J., Lake, A. A., Araujo-Soares, V., White, M., & Summerbell, C. (2016) · Systematic Reviews · doi:10.1186/s13643-016-0337-y · Cited by 486
Metadata
- Authors
- MacDonald, H., Comer, C., Foster, M., Labelle, P. R., Marsalis, S., Nyhan, K., Premji, Z., Rogers, M., Splenda, R., Stansfield, C., & Young, S.
- Venue
- Campbell Systematic Reviews
- Publisher
- Wiley
- Published
- 2024-09
- Volume
- 20
- Issue
- 3
- Pages
- e1433-e1433
- ISSN
- 1891-1803 (ISSN-L)
- OpenAlex
- W4402418527
- Cited by
- 98
- References
- 38
- FWCI
- 5.63
- Type
- journal article
- Language
- en
- Open access
- diamond
- Licence
- creativecommons.org/licenses/by/4.0/ (vor)
- journals.sagepub.com/page/policies/text-and-data-mining-license (tdm)
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.