Supplementary search methods were more effective and offered better value than bibliographic database searching: A case study from public health and environmental enhancement
Open accessThe DOI matches its Crossref record; 15 of the 15 references it cites were found in a registry or an index.
Abstract
Background: We undertook a systematic review to evaluate the health benefits of environmental enhancement and conservation activities. We were concerned that a conventional process of study identification, focusing on exhaustive searches of bibliographic databases as the primary search method, would be ineffective, offering limited value. The focus of this study is comparing study identification methods. We compare (1) an approach led by searches of bibliographic databases with (2) an approach led by supplementary search methods. We retrospectively assessed the effectiveness and value of both approaches. Methods: Effectiveness was determined by comparing (1) the total number of studies identified and screened and (2) the number of includable studies uniquely identified by each approach. Value was determined by comparing included study quality and by using qualitative sensitivity analysis to explore the contribution of studies to the synthesis. Results: The bibliographic databases approach identified 21 409 studies to screen and 2 included qualitative studies were uniquely identified. Study quality was moderate, and contribution to the synthesis was minimal. The supplementary search approach identified 453 studies to screen and 9 included studies were uniquely identified. Four quantitative studies were poor quality but made a substantive contribution to the synthesis; 5 studies were qualitative: 3 studies were good quality, one was moderate quality, and 1 study was excluded from the synthesis due to poor quality. All 4 included qualitative studies made significant contributions to the synthesis. Conclusions: This case study found value in aligning primary methods of study identification to maximise location of relevant evidence.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Methods for the thematic synthesis of qualitative research in systematic reviews
Thomas, J., & Harden, A. (2008) · BMC Medical Research Methodology · doi:10.1186/1471-2288-8-45 · Cited by 9,312
Armijo‐Olivo, S., Stiles, C. R., Hagen, N. A., Biondo, P. D., & Cummings, G. G. (2012) · Journal of Evaluation in Clinical Practice · doi:10.1111/j.1365-2753.2010.01516.x · Cited by 1,539 · cited 1×
“Quality assessment of studies included in a review examines the risk of bias in studies using quantitative study designs, and subjective interpretation in qualitative studies, and the impact on results,(23) guiding the interpretation of findings.(24) In this way, study quality is integral to interpreting the value of…”
BackgroundEgger, M., Jüni, P., Bartlett, C., Holenstein, F., & Sterne, J. (2003) · Health Technology Assessment · doi:10.3310/hta7010 · Cited by 1,181 · cited 3×
“Contribution to the synthesis Quantitative Comprehensive study identification is an important part of evaluating intervention effectiveness as it is linked to producing a reliable estimate of intervention effectiveness (Egger et al., 2003).”
BackgroundResultInfluentialGodin, K., Stapleton, J., Kirkpatrick, S. I., Hanning, R. M., & Leatherdale, S. T. (2015) · Systematic Reviews · doi:10.1186/s13643-015-0125-0 · Cited by 959
Lewin, S., Glenton, C., Munthe-Kaas, H., Carlsen, B., Colvin, C. J., Gülmezoglu, M., Noyes, J., Booth, A., Garside, R., & Rashidian, A. (2015) · PLOS Medicine · doi:10.1371/journal.pmed.1001895 · Cited by 866
Self-citationSearching for grey literature for systematic reviews: challenges and benefits
Mahood, Q., Van Eerd, D., & Irvin, E. (2014) · Research Synthesis Methods · doi:10.1002/jrsm.1106 · Cited by 814 · cited 1×
“Facing similarly challenging searches, other researchers have explored the successful adaptation of conventional search methods to the identification of studies within disparate bodies of grey literature (Adams et al., 2016, Godin et al., 2015, Mahood et al., 2014).”
Adams, J., Hillier-Brown, F. C., Moore, H. J., Lake, A. A., Araujo-Soares, V., White, M., & Summerbell, C. (2016) · Systematic Reviews · doi:10.1186/s13643-016-0337-y · Cited by 486 · cited 3×
“The process of contacting organisations and web‐searching is time intensive(11,57) with accompanying problems of data management and replicability.(11) Bibliographic databases, almost without exception in this review, have export facilities to bibliographic management tools, whereas managing and de‐duplicating studies…”
BackgroundPapaioannou, D., Sutton, A., Carroll, C., Booth, A., & Wong, R. (2010) · Health Information & Libraries Journal · doi:10.1111/j.1471-1842.2009.00863.x · Cited by 318 · cited 2×
“Studies evaluating the use of supplementary search methods were useful in informing this discussion.(21) The study identification protocols are outlined in Figure 2.”
MethodSelf-citationShades of grey: Two forms of grey literature important for reviews in conservation
Haddaway, N. R., & Bayliss, H. R. (2015) · Biological Conservation · doi:10.1016/j.biocon.2015.08.018 · Cited by 180 · cited 6×
“…review, namely: a relative absence of studies using controlled o r otherwise ‘higher order’ st udy designs (Stewart et al., 2005, Fazey et al., 2004, Haddaway and Bayliss, 2015); a difficulty in accessing primary studies to review, due to: delays in publication, limited publication, or simply no…”
BackgroundMethodResultInfluentialHusk, K., Lovell, R., Cooper, C., Stahl-Timmins, W., & Garside, R. (2016) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd010351.pub2 · Cited by 128 · cited 7×
“The synthesis of qualitative studies as reported in our Cochrane Review was used.(20) Once each paper's contribution to the overarching and sub‐themes was identified in the synthesis, we determined which studies were (1) fundamental and necessary to the specific overarching and/or sub‐theme (we term these key studies)…”
BackgroundMethodInfluentialSelf-citation
Research works printed but not linked: 42
- 10.1111/conl.12046 · DOI
- 10.1111/j.1523-1739.2001.99499.x · doi:10.1111/j.1523-1739.2001.99499.x · Paper page
- 10.1017/S0376892904001560 · DOI · Paper page
- 10.1016/j.biocon.2005.06.003 · DOI
- 10.1186/2047-2382-3-14 · DOI · Paper page
- 10.1016/j.envsci.2014.05.010 · DOI · Paper page
- 10.1038/420015a · DOI
- Lefebvre C, 2011, Cochrane Handbook for Systematic Reviews of Interventions Version 5.1.0
- 10.1002/14651858.CD010351 · DOI
- Armstrong RWE, 2011, Cochrane Handbook for Systematic Reviews of Interventions
- 10.1016/S0749-3797(00)00135-5 · DOI
- 10.1080/13511610.2013.777270 · DOI · Paper page
- Sterne JAC, 2011, Cochrane Handbook for Systematic Reviews of Intervention
- 10.1332/0305573042009444 · DOI · Paper page
- 10.1108/17465729200700018 · DOI · Paper page
- 10.1080/1354983042000309289 · DOI
- BTCV, 2010, Wellbeing Comes Naturally: Year One Report
- Eastaugh K, 2010, Wye Wood Evaluation 2006–2009
- 10.1080/02508281.2003.11081414 · DOI · Paper page
- Small Woods A, 2011, Amazon Woman Hereford SF36 Analysis
- Wilson N, 2009, Branching Out. Greenspace and Conservation on Referral
- Townsend M, 2004, Exploration of the Health and Well‐being Benefits of Membership of Truganina Explosives Reserve Preservation Society
- 10.1007/s10393-006-0070-4 · DOI
- Barton J, 2009, The Effects of Green Exercise on Psychological Health and Well‐Being
- 10.1177/030802260506800602 · DOI
- Carter C, 2008, Offenders and Nature Schemes: Using Conservation and Forest Management in Rehabilitation
- O'Brien L, 2010, Volunteering in nature as a way of enabling people to re‐integrate into society, Perspect Public Health
- O'Brien L, 2008, ‘I like to think when I'm gone I will have left this a better place’ Environmental volunteering: motivations, barriers and benefits
- 10.1093/geront/gnq007 · DOI · Paper page
- Reynolds V, 1999, The Green Gym Evaluation of a Pilot Project in Sonning Common, Oxfordshire
- 10.1016/j.ufug.2006.02.001 · DOI
- 10.1186/s12889-015-2214-3 · DOI
- 10.1186/2046-4053-3-135 · DOI · Paper page
- 10.1017/S0266462306051105 · DOI · Paper page
- 10.1542/peds.84.2.374 · DOI · Paper page
- 10.1136/bmj.317.7172.1562 · DOI · Paper page
- 10.1186/2046-4053-3-107 · DOI · Paper page
- 10.4081/jphr.2014.177 · DOI · Paper page
- 10.1186/2046-4053-1-19 · DOI · Paper page
- 10.1017/S0266462310000966 · DOI · Paper page
- Helmer D, 2001, Evidence‐based practice: extending the search to find material for the systematic review, Bull Med Libr Assoc, 89, 346
- 10.1002/jrsm.1036 · DOI · Paper page
Cited by
Haddaway, N. R., Grainger, M. J., & Gray, C. T. (2022) · Research Synthesis Methods · doi:10.1002/jrsm.1563 · Cited by 604
“Citation chasing is a useful supplementary source of information in systematic searches inherent to rigorous evidence syntheses.(3) However, currently there is little evidence that it is being used in rigorous systematic reviews conducted according to the standards of leading evidence synthesis coordinating bodies lik…”
Cooper, C., Booth, A., Varley-Campbell, J., Britten, N., & Garside, R. (2018) · BMC Medical Research Methodology · doi:10.1186/s12874-018-0545-3 · Cited by 561
Self-citationMeeting the review family: exploring review types and associated information retrieval requirements
Sutton, A., Clowes, M., Preston, L., & Booth, A. (2019) · Health Information & Libraries Journal · doi:10.1111/hir.12276 · Cited by 570
Self-citationWhat approaches to social prescribing work, for whom, and in what circumstances? A realist review
Husk, K., Blockley, K., Lovell, R., Bethel, A., Lang, I., Byng, R., & Garside, R. (2020) · Health & Social Care in the Community · doi:10.1111/hsc.12839 · Cited by 381
Self-citationArundell, L.-L., Greenwood, H., Baldwin, H., Kotas, E., Smith, S., Trojanowska, K., & Cooper, C. (2020) · Systematic Reviews · doi:10.1186/s13643-020-01333-6 · Cited by 53
“The purpose was to determine suitable research questions and use these to develop the approach to study identification, as well as pilot inclusion criteria and data extraction for the mapping review [23, 24].”
MethodSelf-citation
Metadata
- Authors
- Cooper, C., Lovell, R., Husk, K., Booth, A., & Garside, R.
- Venue
- Research Synthesis Methods
- Publisher
- Wiley
- Published
- 2017-12-26
- Volume
- 9
- Issue
- 2
- Pages
- 195-223
- ISSN
- 1759-2879 (ISSN-L)
- 1759-2887
- OpenAlex
- W2768600157
- Cited by
- 66
- References
- 15
- Type
- supplementary materials
- Language
- en
- Open access
- green
- Funders
- NIHR School for Public Health Research (SPHR)
- National Institute for Health Research (NIHR) Collaboration for Leadership in Applied Health Research and Care South West Peninsula
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Supplementary search methods were more effective and offered better value than bibliographic database searching: A case study from public health and environmental enhancement. https://app.cochat.ai/papers/10.1002/jrsm.1286
- The paper
- Cooper, C., Lovell, R., Husk, K., Booth, A., & Garside, R. (2018). Supplementary search methods were more effective and offered better value than bibliographic database searching: A case study from public health and environmental enhancement. Research Synthesis Methods. https://doi.org/10.1002/jrsm.1286
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.