Appraising qualitative research for inclusion in systematic reviews: a quantitative and qualitative comparison of three methods
Abstract
Objective: Qualitative research is increasingly valued as part of the evidence for policy and practice, but how it should be appraised is contested. Various appraisal methods, including checklists and other structured approaches, have been proposed but rarely evaluated. We aimed to compare three methods for appraising qualitative research papers that were candidates for inclusion in a systematic review of evidence on support for breast-feeding. Method: A sample of 12 research papers on support for breast-feeding was appraised by six qualitative reviewers using three appraisal methods: unprompted judgement, based on expert opinion; a UK Cabinet Office quality framework; and CASP, a Critical Appraisal Skills Programme tool. Papers were assigned, following appraisals, to 1 of 5 categories, which were dichotomized to indicate whether or not papers should be included in a systematic review. Patterns of agreement in categorization of papers were assessed quantitatively using κ statistics, and qualitatively using cross-case analysis. Results: Agreement in categorizing papers across the three methods was slight ( κ =0.13; 95% CI 0.06–0.24). Structured approaches did not appear to yield higher agreement than that by unprompted judgement. Qualitative analysis revealed reviewers' dilemmas in deciding between the potential impact of findings and the quality of the research execution or reporting practice. Structured instruments appeared to make reviewers more explicit about the reasons for their judgements. Conclusions: Structured approaches may not produce greater consistency of judgements about whether to include qualitative papers in a systematic review. Future research should address how appraisals of qualitative research should be incorporated in systematic reviews.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Patients with MND can relate to peers who are in a similar situation.
“Patients could relate to peers who were in a similar situation (2, 16, 17, 19, 25, 29).”
From Patients experiences of maintaining mental well-being and hope within motor neuron disease: a thematic synthesis (2015) · cites this paper as [16]
Interactions with others are beneficial for patients with MND because they provide a chance for individuals to give back or care for others.
“Apart from the benefits of being connected and valued with others, interactions were also beneficial in that they provided a chance for individuals to give back or care for others (5, 16, 17, 18, 19, 27).”
From Patients experiences of maintaining mental well-being and hope within motor neuron disease: a thematic synthesis (2015) · cites this paper as [16]
Patients with MND used alternative therapies, adapted health behaviors including rehabilitative exercises, considering care routines, or undertaking hobbies or activities, and made dietary adaptations.
“This included, using alternative therapies (2, 11, 16), adapting health behaviors including rehabilitative exercises, considering care routines, or undertaking hobbies or activities (5, 11, 16) and making dietary adaption (2, 9, 16).”
From Patients experiences of maintaining mental well-being and hope within motor neuron disease: a thematic synthesis (2015) · cites this paper as [16]
Other studies document the value of humor for patients with MND, including black humor or teasing.
“Other studies documented the value of humor for patients (4, 11, 16, 18, 20, 29), this included black humor or teasing.”
From Patients experiences of maintaining mental well-being and hope within motor neuron disease: a thematic synthesis (2015) · cites this paper as [16]
A number of studies highlighted that patients with MND require physical assistance in increasing quantities.
“A number of studies highlighted that patients required physical assistance in increasing quantities (5, 11, 16, 18).”
From Patients experiences of maintaining mental well-being and hope within motor neuron disease: a thematic synthesis (2015) · cites this paper as [16]
Cited by
Sorted by citations
Enhancing transparency in reporting the synthesis of qualitative research: ENTREQ
Tong, A., Flemming, K., McInnes, E., Oliver, S., & Craig, J. (2012) · BMC Medical Research Methodology · doi:10.1186/1471-2288-12-181 · Cited by 3,569 · see citing works
“Also, there is little evidence on how the quality of reporting reflects the robustness, trustworthiness and transferability of the findings of qualitative studies [35].”
BackgroundPerski, O., Blandford, A., West, R., & Michie, S. (2017) · Translational Behavioral Medicine · doi:10.1007/s13142-016-0453-1 · Cited by 1,210 · see citing works
Long, H. A., French, D. P., & Brooks, J. M. (2020) · Research Methods in Medicine & Health Sciences · doi:10.1177/2632084320947559 · Cited by 1,112 · see citing works
Patient Adherence to Tuberculosis Treatment: A Systematic Review of Qualitative Research
Munro, S. A., Lewin, S. A., Smith, H. J., Engel, M. E., Fretheim, A., & Volmink, J. (2007) · PLoS Medicine · doi:10.1371/journal.pmed.0040238 · Cited by 1,080 · see citing works
Lewin, S., Glenton, C., Munthe-Kaas, H., Carlsen, B., Colvin, C. J., Gülmezoglu, M., Noyes, J., Booth, A., Garside, R., & Rashidian, A. (2015) · PLOS Medicine · doi:10.1371/journal.pmed.1001895 · Cited by 866 · see citing works
Self-citationO’Connor, S., Hanlon, P., O’Donnell, C. A., Garcia, S., Glanville, J., & Mair, F. S. (2016) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-016-0359-3 · Cited by 636 · see citing works
Feedback from research participants: are member checks useful in qualitative research?
Thomas, D. R. (2017) · Qualitative Research in Psychology · doi:10.1080/14780887.2016.1219435 · Cited by 487 · see citing works
Carroll, C., Booth, A., & Cooper, K. (2011) · BMC Medical Research Methodology · doi:10.1186/1471-2288-11-29 · Cited by 461 · see citing works
Self-citationUse of communities of practice in business and health care sectors: A systematic review
Li, L. C., Grimshaw, J. M., Nielsen, C., Judd, M., Coyte, P. C., & Graham, I. D. (2009) · Implementation Science · doi:10.1186/1748-5908-4-27 · Cited by 377 · see citing works
Malpass, A., Shaw, A., Sharp, D., Walter, F., Feder, G., Ridd, M., & Kessler, D. (2009) · Social Science & Medicine · doi:10.1016/j.socscimed.2008.09.068 · Cited by 368 · see citing works
Carroll, C., Booth, A., & Lloyd-Jones, M. (2012) · Qualitative Health Research · doi:10.1177/1049732312452937 · Cited by 248 · see citing works
Self-citationCarroll, C., & Booth, A. (2015) · Research Synthesis Methods · doi:10.1002/jrsm.1128 · Cited by 235 · see citing works
Self-citationAnderson, A. H., Stephenson, J., & Carter, M. (2017) · Research in Autism Spectrum Disorders · doi:10.1016/j.rasd.2017.04.002 · Cited by 196 · see citing works
Newton-Levinson, A., Leichliter, J. S., & Chandra-Mouli, V. (2016) · Journal of Adolescent Health · doi:10.1016/j.jadohealth.2016.03.014 · Cited by 172 · see citing works
Demain, S., Gonçalves, A.-C., Areia, C., Oliveira, R., Marcos, A. J., Marques, A., Parmar, R., & Hunt, K. (2015) · PLOS ONE · doi:10.1371/journal.pone.0125457 · Cited by 173 · see citing works
Patients’ experiences of chronic non-malignant musculoskeletal pain: a qualitative systematic review
Toye, F., Seers, K., Allcock, N., Briggs, M., Carr, E., Andrews, J., & Barker, K. (2013) · British Journal of General Practice · doi:10.3399/bjgp13x675412 · Cited by 156 · see citing works
Santiago-Delefosse, M., Gavin, A., Bruchez, C., Roux, P., & Stephen, S. L. (2016) · Social Science & Medicine · doi:10.1016/j.socscimed.2015.11.007 · Cited by 146 · see citing works
Hughes-Morley, A., Young, B., Waheed, W., Small, N., & Bower, P. (2015) · Journal of Affective Disorders · doi:10.1016/j.jad.2014.10.005 · Cited by 132 · see citing works
Self-citationBray, I., Reece, R., Sinnett, D., Martin, F., & Hayward, R. (2022) · Environmental Research · doi:10.1016/j.envres.2022.114081 · Cited by 124 · see citing works
Fernandes, L. G., Devan, H., Fioratti, I., Kamper, S. J., Williams, C. M., & Saragiotto, B. T. (2022) · Pain · doi:10.1097/j.pain.0000000000002364 · Cited by 118 · see citing works
Malpass, A., Carel, H., Ridd, M., Shaw, A., Kessler, D., Sharp, D., Bowden, M., & Wallond, J. (2012) · Mindfulness · doi:10.1007/s12671-011-0081-2 · Cited by 112 · see citing works
Patterns of unmet needs in adolescent and young adult (AYA) cancer survivors: in their own words
Wong, A. W. K., Chang, T.-T., Christopher, K., Lau, S. C. L., Beaupin, L. K., Love, B., Lipsey, K. L., & Feuerstein, M. (2017) · Journal of Cancer Survivorship · doi:10.1007/s11764-017-0613-4 · Cited by 109 · see citing works
A meta-ethnography of patients’ experience of chronic non-malignant musculoskeletal pain
Toye, F., Seers, K., Allcock, N., Briggs, M., Carr, E., Andrews, J., & Barker, K. (2013) · Health Services and Delivery Research · doi:10.3310/hsdr01120 · Cited by 98 · see citing works
Robson, R. C., Pham, B., Hwee, J., Thomas, S. M., Rios, P., Page, M. J., & Tricco, A. C. (2019) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2018.10.003 · Cited by 86 · see citing works
Klemanski, D. L., Browning, K. K., & Kue, J. (2016) · Journal of Cancer Survivorship · doi:10.1007/s11764-015-0452-0 · Cited by 75 · see citing works
Health Care Workers’ Experiences of Mindfulness Training: a Qualitative Review
Morgan, P., Simpson, J., & Smith, A. (2015) · Mindfulness · doi:10.1007/s12671-014-0313-3 · Cited by 75 · see citing works
Toye, F., Seers, K., Allcock, N., Briggs, M., Carr, E., & Barker, K. (2016) · Disability and Rehabilitation · doi:10.3109/09638288.2015.1049377 · Cited by 62 · see citing works
Hennelly, N., Cooney, A., Houghton, C., & O’Shea, E. (2021) · The Gerontologist · doi:10.1093/geront/gnz159 · Cited by 61 · see citing works
South, J., Bagnall, A.-M., Hulme, C., Woodall, J., Longo, R., Dixey, R., Kinsella, K., Raine, G., Vinall-Collier, K., & Wright, J. (2014) · Health Services and Delivery Research · doi:10.3310/hsdr02350 · Cited by 63 · see citing works
Reeve, A., Tickle, A., & Moghaddam, N. (2018) · Mental Health Review Journal · doi:10.1108/mhrj-11-2017-0052 · Cited by 49 · see citing works
Davies, L., Beard, D., Cook, J. A., Price, A., Osbeck, I., & Toye, F. (2021) · Trials · doi:10.1186/s13063-021-05403-5 · Cited by 41 · see citing works
Soundy, A., & Condon, N. (2015) · Frontiers in Psychology · doi:10.3389/fpsyg.2015.00606 · Cited by 35 · see citing works
“A fatal flaw is defined as a methodological weakness that compromises the trustworthiness of the data (Dixon-Woods et al., 2007).”
BackgroundPradelli, L., Risoli, C., Summer, E., Bellini, G., Mozzarelli, F., Anderson, G., Guasconi, M., Artioli, G., Bonacaro, A., & Sarli, L. (2025) · BMJ Open · doi:10.1136/bmjopen-2024-087268 · Cited by 42 · see citing works
Little, H., Tickle, A., & das Nair, R. (2018) · Psychology and Psychotherapy Theory Research and Practice · doi:10.1111/papt.12156 · Cited by 28 · see citing works
Pickles, K., Eassey, D., Reddel, H. K., Locock, L., Kirkpatrick, S., & Smith, L. (2018) · Health Expectations · doi:10.1111/hex.12605 · Cited by 28 · see citing works
Saragosa, M., Jeffs, L., Okrainec, K., & Kuluski, K. (2022) · Dementia · doi:10.1177/14713012211031779 · Cited by 21 · see citing works
Meek, C., Topcu, G., Moghaddam, N., & das Nair, R. (2021) · Disability and Rehabilitation · doi:10.1080/09638288.2020.1734105 · Cited by 10 · see citing works
Cooper, A. M., Coffey, M., & Dugdill, L. (2014) · Pragmatic and Observational Research · doi:10.2147/por.s52287 · Cited by 8 · see citing works
“Few studies are using the available reporting guidelines, eg, the CONSORT statement for RCTs(41,42) and cluster RCTs,(43,44) the QUORUM statement for meta-analyses,(45) the TREND statement for nonrandomized evaluations,(46) the STROBE statement for observational epidemiological studies,(47) and other methods for quali…”
- Authors
- Dixon-Woods, M., Sutton, A., Shaw, R., Miller, T., Smith, J., Young, B., Bonas, S., Booth, A., & Jones, D.
- Affiliations
- Department of Health Sciences, University of Leicester, Leicester, UK
- University of Leicester (GB)
- Aston University (GB)
- Oxford Brookes University (GB)
- University of London (GB)
- University of Liverpool (GB)
- University of Sheffield (GB)
- Venue
- Journal of Health Services Research & Policy
- Publisher
- SAGE Publications
- Published
- 2007-01-01
- Volume
- 12
- Issue
- 1
- Pages
- 42-47
- ISSN
- 1355-8196 (ISSN-L)
- 1758-1060
- OpenAlex
- W2077442888
- Cited by
- 546
- References
- 0
- Citations by year
- 2026: 12 · 2025: 26 · 2024: 15 · 2023: 24 · 2022: 41 · 2021: 45 · 2020: 32 · 2019: 38 · 2018: 44 · 2017: 40 · 2016: 36 · 2015: 39 · 2014: 31 · 2013: 21 · 2012: 24
- FWCI
- 3.34
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Journal of Health Services Research & Policy (publishedVersion)
- PubMed (publishedVersion)
- BIROn (Birkbeck, University of London) (submittedVersion)
- Leicester Research Archive (University of Leicester) (submittedVersion)
- Funders
- Economic and Social Research Council (RES-333-34-3002)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Education › Qualitative Research Methods and Applications
- Social Sciences › Sociology and Political Science › Delphi Technique in Research
- Keywords
- evaluation checklist, CASP, interobserver agreement, expert opinion, research quality assessment
- MeSH terms
- Evaluation Studies as Topic, United Kingdom, Peer Review, Research, Qualitative Research
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
