The role of systematic reviews of qualitative evidence in evaluating interventions: a case study
Open accessThe DOI matches its Crossref record; 14 of the 14 references it cites were found in a registry or an index.
Abstract
Systematic reviews of qualitative evidence have been widely used to provide information on the context and implementation of interventions, and their potential barriers and facilitators. However, such reviews face a number of methodological challenges, and there are ongoing debates as to how qualitative evidence can best be used to inform our understanding of interventions. In this paper, we use a case study of two systematic reviews of qualitative evidence on the prevention of skin cancer to explore these issues. We find that qualitative evidence not directly related to interventions is likely to be of value for such reviews, that it is often not possible to construct fully comprehensive search strategies, and that there are diminishing returns to the synthesis, in terms of added value or insight, from the inclusion of large numbers of primary studies. We conclude that there are a number of ways in which systematic reviews of qualitative evidence can be utilised in conjunction with evidence on intervention effectiveness, without compromising the rigour of the review process. In particular, the use of theory to inform frameworks for synthesis is a promising way to integrate a broader range of qualitative evidence. Copyright © 2012 John Wiley & Sons, Ltd.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The Discovery of Grounded Theory; Strategies for Qualitative Research
Glaser, B. G., Strauss, A. L., & Strutzel, E. (1968) · Nursing Research · doi:10.1097/00006199-196807000-00014 · Cited by 39,449
Methods for the thematic synthesis of qualitative research in systematic reviews
Thomas, J., & Harden, A. (2008) · BMC Medical Research Methodology · doi:10.1186/1471-2288-8-45 · Cited by 9,312
Realist review - a new method of systematic review designed for complex policy interventions
Pawson, R., Greenhalgh, T., Harvey, G., & Walshe, K. (2005) · Journal of Health Services Research & Policy · doi:10.1258/1355819054308530 · Cited by 3,190 · cited 1×
“Such an approach has similarities to realist review methodology, where papers are selected based on rigour and relevance to the research question (Pawson et al., 2005).”
BackgroundUsing meta ethnography to synthesise qualitative research: a worked example
Britten, N., Campbell, R., Pope, C., Donovan, J., Morgan, M., & Pill, R. (2002) · Journal of Health Services Research & Policy · doi:10.1258/135581902320432732 · Cited by 1,110 · cited 1×
“For example, in meta-ethnography, the conceptually richer papers have greater weight in the synthesis, with other papers merely illustrating these concepts with descriptive themes (Britten et al., 2002; Garside et al., 2008).”
ResultPopay, J., Rogers, A., & Williams, G. (1998) · Qualitative Health Research · doi:10.1177/104973239800800305 · Cited by 987
How can systematic reviews incorporate qualitative research? A critical perspective
Dixon-Woods, M., Bonas, S., Booth, A., Jones, D. R., Miller, T., Sutton, A. J., Shaw, R. L., Smith, J. A., & Young, B. (2006) · Qualitative Research · doi:10.1177/1468794106058867 · Cited by 860
Glaser, B. G., & Holton, J. (2004) · Forum Qualitative Sozialforschung / Forum: Qualitative Social Research · doi:10.17169/fqs-5.2.607 · Cited by 783
Integrating qualitative research with trials in systematic reviews
Thomas, J., Harden, A., Oakley, A., Oliver, S., Sutcliffe, K., Rees, R., Brunton, G., & Kavanagh, J. (2004) · BMJ · doi:10.1136/bmj.328.7446.1010 · Cited by 515 · cited 1×
“…of SRs of qualitative evidence in evaluating interventions may include explaining heterogeneity in the findings of quantitative evaluation research (Thomas et al., 2004), providing indicative prima facie evidence of the likelihood of intervention effectiveness in areas where quantitative outcome…”
BackgroundCarroll, C., Booth, A., & Cooper, K. (2011) · BMC Medical Research Methodology · doi:10.1186/1471-2288-11-29 · Cited by 461 · cited 1×
“Both reviews used the Health Belief Model (HBM) as a framework for analysis; they might therefore be considered ‘framework syntheses’ (cf. Carroll et al., 2011; Lorenc et al., 2008), although the framework was drawn from the included studies rather than fixed a priori.”
MethodUsing logic models to capture complexity in systematic reviews: Logic Models in Systematic Reviews
Anderson, L. M., Petticrew, M., Rehfuess, E., Armstrong, R., Ueffing, E., Baker, P., Francis, D., & Tugwell, P. (2011) · Research Synthesis Methods · doi:10.1002/jrsm.32 · Cited by 372 · cited 1×
“…‘open up the black box’ of outcome evaluation, build more adequate theories of intervention effectiveness, and provide more useful information to assist practitioners and policymakers in implementing interventions (Anderson et al., 2011; Noyes et al., 2011; Oliver et al., 2005; Popay et al., 1998).”
BackgroundHarden, A., Garcia, J., Oliver, S., Rees, R., Shepherd, J., Brunton, G., & Oakley, A. (2004) · Journal of Epidemiology and Community Health · doi:10.1136/jech.2003.014829 · Cited by 328 · cited 2×
“…inadequate (Whitehead et al., 2004), enabling researchers to engage a wider range of stakeholders by giving a richer account of the available evidence (Popay et al., 1998), and facilitating a greater understanding of the views of those affected by interventions (Harden et al., 2004).”
BackgroundOliver, S., Harden, A., Rees, R., Shepherd, J., Brunton, G., Garcia, J., & Oakley, A. (2005) · Evaluation · doi:10.1177/1356389005059383 · Cited by 221
Applications of text mining within systematic reviews
Thomas, J., McNaught, J., & Ananiadou, S. (2011) · Research Synthesis Methods · doi:10.1002/jrsm.27 · Cited by 212 · cited 2×
“…viability of text-mining techniques may provide ways to make this process more efficient, by using techniques such as automated document clustering to provide an initial overview of the available evidence base across a broad range of topic areas (Ananadiou et al., 2009; Thomas et al., 2011).”
MethodRoen, K., Arai, L., Roberts, H., & Popay, J. (2006) · Social Science & Medicine · doi:10.1016/j.socscimed.2006.02.013 · Cited by 106 · cited 1×
“…inadequate (Whitehead et al., 2004), enabling researchers to engage a wider range of stakeholders by giving a richer account of the available evidence (Popay et al., 1998), and facilitating a greater understanding of the views of those affected by interventions (Harden et al., 2004).”
Background
Cited by
“Best fit” framework synthesis: refining the method
Carroll, C., Booth, A., Leaviss, J., & Rick, J. (2013) · BMC Medical Research Methodology · doi:10.1186/1471-2288-13-37 · Cited by 649
“conceptual models [62]: All of the included studies repre-”
BackgroundMethodBooth, A. (2016) · Systematic Reviews · doi:10.1186/s13643-016-0249-x · Cited by 569
Cooper, C., Booth, A., Varley-Campbell, J., Britten, N., & Garside, R. (2018) · BMC Medical Research Methodology · doi:10.1186/s12874-018-0545-3 · Cited by 561
Self-citationFlemming, K., Booth, A., Garside, R., Tunçalp, Ö., & Noyes, J. (2019) · BMJ Global Health · doi:10.1136/bmjgh-2018-000882 · Cited by 389
Self-citationBrown, B., Gude, W. T., Blakeman, T., van der Veer, S. N., Ivers, N., Francis, J. J., Lorencatto, F., Presseau, J., Peek, N., & Daker-White, G. (2019) · Implementation Science · doi:10.1186/s13012-019-0883-5 · Cited by 387
Metadata
- Authors
- Lorenc, T., Pearson, M., Jamal, F., Cooper, C., & Garside, R.
- Venue
- Research Synthesis Methods
- Publisher
- Wiley
- Published
- 2012-03
- Volume
- 3
- Issue
- 1
- Pages
- 1-10
- ISSN
- 1759-2879 (ISSN-L)
- 1759-2887
- OpenAlex
- W2118876890
- Cited by
- 61
- References
- 14
- FWCI
- 1.42
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.