Optimising the value of the critical appraisal skills programme (CASP) tool for quality appraisal in qualitative evidence synthesis
Open accessThe DOI matches its Crossref record; 49 of the 49 references it cites were found in a registry or an index.
Abstract
The value of qualitative evidence synthesis for informing healthcare policy and practice within evidence-based medicine is increasingly recognised. However, there is a lack of consensus regarding how to judge the methodological quality of qualitative studies being synthesised and debates around the extent to which such assessment is possible and appropriate. The Critical Appraisal Skills Programme (CASP) tool is the most commonly used tool for quality appraisal in health-related qualitative evidence syntheses, with endorsement from the Cochrane Qualitative and Implementation Methods Group. The tool is recommended for novice qualitative researchers, but there is little existing guidance on its application. This article considers issues related to the suitability and usability of the CASP tool for quality appraisal in qualitative evidence synthesis in order to support and improve future appraisal exercises framed by the tool. We reflect on our practical experience of using the tool in a systematic review and qualitative evidence synthesis. We discuss why it is worth considering a study’s underlying theoretical, ontological and epistemological framework and how this could be incorporated into the tool by way of a novel question. We consider how particular features of the tool may impact its interpretation, the appraisal results and the subsequent synthesis. We discuss how to use quality appraisal results to inform the next stages of evidence synthesis and present a novel approach to organising the synthesis, whereby studies deemed to be of higher quality contribute relatively more to the synthesis. We propose tool modifications, user guidance, and areas for future methodological research.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“This indicates that included studies were primarily of moderate to high quality [64, 65].”
From Barriers and facilitators to diet, physical activity and lifestyle behavior intervention adherence: a qualitative systematic review of the literature (2023) · cites this paper as [65]
In the citing paper
“The qualitative research appraisal (Sec. 4.3) of each paper was done collaboratively (details in Appx. B) with the modified 11-question CASP checklist [35], and detailed justifications are provided as Supplementary Materials.”
From Generative AI in Game Development: A Qualitative Research Synthesis (2026) · cites this paper as [35]
In the citing paper
“Appraisal was conducted with the CASP, which has been previously endorsed by the Cochrane Qualitative and Implementation Methods Group and widely used in health-related meta-syntheses (Butler et al., 2020;Long et al., 2020).”
From Fathers’ experiences of perinatal death following miscarriage, stillbirth, and neonatal death: A meta-ethnography (2026) · cites this paper as Long et al., 2020
In the citing paper
“The methodological quality of all papers was assessed using the Critical Appraisal Skills Programme (CASP) checklist for qualitative research [64, 65].”
From Barriers and facilitators to diet, physical activity and lifestyle behavior intervention adherence: a qualitative systematic review of the literature (2023) · cites this paper as [65]
In the citing paper
“appraising primary study quality with the Critical Appraisal Skills Programme (CASP) [35].”
From Generative AI in Game Development: A Qualitative Research Synthesis (2026) · cites this paper as [35]
4 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Lincoln, Y. S., Guba, E. G., & Pilotta, J. J. (1985) · International Journal of Intercultural Relations · doi:10.1016/0147-1767(85)90062-8 · Cited by 45,787
Tong, A., Sainsbury, P., & Craig, J. (2007) · International Journal for Quality in Health Care · doi:10.1093/intqhc/mzm042 · Cited by 42,509 · cited 1×
“There have been recent calls for improved reporting of qualitative research.(49) In response, qualitative study reporting standards have been developed, such as APA JARS (for qualitative psychology research),(45) Consolidated criteria for Reporting Qualitative research (COREQ; for qualitative interview and focus group…”
Reflecting on reflexive thematic analysis
Braun, V., & Clarke, V. (2019) · Qualitative Research in Sport, Exercise and Health · doi:10.1080/2159676x.2019.1628806 · Cited by 19,677
Methods for the thematic synthesis of qualitative research in systematic reviews
Thomas, J., & Harden, A. (2008) · BMC Medical Research Methodology · doi:10.1186/1471-2288-8-45 · Cited by 9,312
Qualitative Psychology: A Practical Guide to Research Methods
(2008) · QMiP Bulletin · doi:10.53841/bpsqmip.2008.1.6.47 · Cited by 4,351
Enhancing transparency in reporting the synthesis of qualitative research: ENTREQ
Tong, A., Flemming, K., McInnes, E., Oliver, S., & Craig, J. (2012) · BMC Medical Research Methodology · doi:10.1186/1471-2288-12-181 · Cited by 3,569 · cited 1×
“In response, qualitative study reporting standards have been developed, such as APA JARS (for qualitative psychology research),(45) Consolidated criteria for Reporting Qualitative research (COREQ; for qualitative interview and focus group studies)(49) and Enhancing transparency in reporting the synthesis of qualitativ…”
MethodA manifesto for reproducible science
Munafò, M. R., Nosek, B. A., Bishop, D. V. M., Button, K. S., Chambers, C. D., Percie du Sert, N., Simonsohn, U., Wagenmakers, E.-J., Ware, J. J., & Ioannidis, J. P. A. (2017) · Nature Human Behaviour · doi:10.1038/s41562-016-0021 · Cited by 3,397
Smith, B., & McGannon, K. R. (2018) · International Review of Sport and Exercise Psychology · doi:10.1080/1750984x.2017.1317357 · Cited by 3,074
Noblit, G., & Hare, R. (1988) · doi:10.4135/9781412985000 · Cited by 2,723
Critical appraisal skills programme
Singh, J. (2013) · Journal of Pharmacology and Pharmacotherapeutics · doi:10.4103/0976-500x.107697 · Cited by 2,556
Evolving guidelines for publication of qualitative research studies in psychology and related fields
Elliott, R., Fischer, C. T., & Rennie, D. L. (1999) · British Journal of Clinical Psychology · doi:10.1348/014466599162782 · Cited by 2,491
Dilemmas in qualitative health research
Yardley, L. (2000) · Psychology & Health · doi:10.1080/08870440008400302 · Cited by 2,310
Levitt, H. M., Bamberg, M., Creswell, J. W., Frost, D. M., Josselson, R., & Suárez-Orozco, C. (2018) · American Psychologist · doi:10.1037/amp0000151 · Cited by 2,286
Rigor or rigor mortis: The problem of rigor in qualitative research revisited
Sandelowski, M. (1993) · Advances in Nursing Science · doi:10.1097/00012272-199312000-00002 · Cited by 1,299
Validity in qualitative research revisited
Cho, J., & Trent, A. (2006) · Qualitative Research · doi:10.1177/1468794106065006 · Cited by 1,037
Levitt, H. M., Motulsky, S. L., Wertz, F. J., Morrow, S. L., & Ponterotto, J. G. (2017) · Qualitative Psychology · doi:10.1037/qup0000082 · Cited by 851
The problem of appraising qualitative research
Dixon-Woods, M. (2004) · Quality and Safety in Health Care · doi:10.1136/qshc.2003.008714 · Cited by 653 · cited 2×
“Others have agreed ‘essential’ criteria that studies must meet,(17,42) assessed the descriptive ‘thinness’ and ‘thickness’ of evidence(43) and suggested establishing ‘fatal flaws’ in the design and conduct of qualitative research.(37) However, there is little academic consensus or empirical evidence on core criteria.”
BackgroundDixon-Woods, M., Sutton, A., Shaw, R., Miller, T., Smith, J., Young, B., Bonas, S., Booth, A., & Jones, D. (2007) · Journal of Health Services Research & Policy · doi:10.1258/135581907779497486 · Cited by 546
Noyes, J., Booth, A., Flemming, K., Garside, R., Harden, A., Lewin, S., Pantoja, T., Hannes, K., Cargo, M., & Thomas, J. (2018) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2017.06.020 · Cited by 480 · cited 5×
“A key stage common to all systematic reviews is quality appraisal of the evidence to be synthesized.(1,8) There is broad debate and little consensus among the academic community over what constitutes ‘quality’ in qualitative research.”
BackgroundMethodInfluentialHannes, K., Lockwood, C., & Pearson, A. (2010) · Qualitative Health Research · doi:10.1177/1049732310378656 · Cited by 409
Sandelowski, M., & Barroso, J. (2002) · International Journal of Qualitative Methods · doi:10.1177/160940690200100107 · Cited by 335
Hannes, K., & Macaitis, K. (2012) · Qualitative Research · doi:10.1177/1468794111432992 · Cited by 320
Evidence-Based Practice Across the Health Professions
Hoffmann, T., Bennett, S. H., & Mar, C. D. (2009) · e-publications@bond (Bond University) · Cited by 294
Noyes, J., & Popay, J. (2007) · Journal of Advanced Nursing · doi:10.1111/j.1365-2648.2006.04092.x · Cited by 272
Synthesizing qualitative research: a review of published reports
Dixon-Woods, M., Booth, A., & Sutton, A. J. (2007) · Qualitative Research · doi:10.1177/1468794107078517 · Cited by 260
Carroll, C., Booth, A., & Lloyd-Jones, M. (2012) · Qualitative Health Research · doi:10.1177/1049732312452937 · Cited by 248 · cited 6×
“The authors of reviews incorporating sensitivity analyses concluded that excluding lower quality studies did not affect their synthesis findings in any meaningful way(17,35,42,43) and believe there is an increasingly strong case for excluding lower quality studies from a synthesis.(17) However, Garside has given an al…”
BackgroundMethodResultInfluentialCarroll, C., & Booth, A. (2015) · Research Synthesis Methods · doi:10.1002/jrsm.1128 · Cited by 235
Dialectical Pluralism: A Metaparadigm Whose Time Has Come
Johnson, R. B. (2017) · Journal of Mixed Methods Research · doi:10.1177/1558689815607692 · Cited by 232
Cochrane Qualitative and Implementation Methods Group guidance series—paper 1: introduction
Noyes, J., Booth, A., Cargo, M., Flemming, K., Garside, R., Hannes, K., Harden, A., Harris, J., Lewin, S., Pantoja, T., et al. (2018) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2017.09.025 · Cited by 177
Harris, J. L., Booth, A., Cargo, M., Hannes, K., Harden, A., Flemming, K., Garside, R., Pantoja, T., Thomas, J., & Noyes, J. (2018) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2017.10.023 · Cited by 165
Harden, A., Thomas, J., Cargo, M., Harris, J., Pantoja, T., Flemming, K., Booth, A., Garside, R., Hannes, K., & Noyes, J. (2017) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2017.11.029 · Cited by 164
Flemming, K., Booth, A., Hannes, K., Cargo, M., & Noyes, J. (2017) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2017.10.022 · Cited by 140
Garside, R. (2014) · Innovation: The European Journal of Social Science Research · doi:10.1080/13511610.2013.777270 · Cited by 134
Metasynthetic Madness: What Kind of Monster Have We Created?
Thorne, S. (2017) · Qualitative Health Research · doi:10.1177/1049732316679370 · Cited by 132
A matter of taste: evaluating the quality of qualitative research
Sandelowski, M. (2015) · Nursing Inquiry · doi:10.1111/nin.12080 · Cited by 112
Munthe‐Kaas, H. M., Glenton, C., Booth, A., Noyes, J., & Lewin, S. A. (2019) · BMC Medical Research Methodology · doi:10.1186/s12874-019-0728-6 · Cited by 104
Cargo, M., Harris, J., Pantoja, T., Booth, A., Harden, A., Hannes, K., Thomas, J., Flemming, K., Garside, R., & Noyes, J. (2017) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2017.11.028 · Cited by 86
Electric fans for reducing adverse health impacts in heatwaves
Gupta, S., Carmichael, C., Simpson, C., Clarke, M. J., Allen, C., Gao, Y., Chan, E. Y. Y., & Murray, V. (2012) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd009888.pub2 · Cited by 84
Malterud, K. (2018) · Qualitative Health Research · doi:10.1177/1049732318795864 · Cited by 79
Long, H. A., Brooks, J. M., Harvie, M., Maxwell, A., & French, D. (2019) · British Journal of Cancer · doi:10.1038/s41416-019-0524-4 · Cited by 72
Self-citationMaking a difference: towards a method for weighing the evidence in a qualitative synthesis
Boeije, H. R., van Wesel, F., & Alisic, E. (2011) · Journal of Evaluation in Clinical Practice · doi:10.1111/j.1365-2753.2011.01674.x · Cited by 67
Enhancing the quality and transparency of qualitative research methods in health psychology
Shaw, R. L., Bishop, F. L., Horwood, J. P., Chilcot, J., & Arden, M. A. (2019) · British Journal of Health Psychology · doi:10.1111/bjhp.12393 · Cited by 67
Critical remarks on the Italian research assessment exercise VQR 2011–2014
Franceschini, F., & Maisano, D. A. (2017) · Journal of Informetrics · doi:10.1016/j.joi.2017.02.005 · Cited by 63
Hannes, K., Booth, A., Harris, J., & Noyes, J. (2013) · Systematic Reviews · doi:10.1186/2046-4053-2-84 · Cited by 61
Dalton, J., Booth, A., Noyes, J., & Sowden, A. J. (2017) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2017.04.020 · Cited by 52
A context-consent meta-framework for designing open (qualitative) data studies
Branney, P., Reid, K., Frost, N. A., Coan, S., Mathieson, A., & Woolhouse, M. (2019) · Qualitative Research in Psychology · doi:10.1080/14780887.2019.1605477 · Cited by 49
Booth, A. (2018) · Qualitative Health Research · doi:10.1177/1049732318808247 · Cited by 29
How to locate and appraise qualitative research in complementary and alternative medicine
Franzel, B., Schwiegershausen, M., Heusser, P., & Berger, B. (2013) · BMC Complementary and Alternative Medicine · doi:10.1186/1472-6882-13-125 · Cited by 24
Excellence in Research in Australia 2010, 2012, and 2015: The Rising of the Curate's Soufflé?
Crowe, S. F., & Watt, S. (2016) · Australian Psychologist · doi:10.1111/ap.12248 · Cited by 11
Research works printed but not linked: 3
- Higgins J, 2008, Cochrane Database Syst Rev, 5, 187
- Braun V, 2012, APA Handb Res Methods Psychol Vol 2 Res Des Quant Qual Neuropsychol Biol, 2, 57
- Winerman L., 2018, Monitor on Psychology, 36
Cited by
Sharkiya, S. H. (2023) · BMC Health Services Research · doi:10.1186/s12913-023-09869-8 · Cited by 416
Consequences of visiting restrictions during the COVID‐19 pandemic: An integrative review
Hugelius, K., Harada, N., & Marutani, M. (2021) · International Journal of Nursing Studies · doi:10.1016/j.ijnurstu.2021.104000 · Cited by 378
Waddell, A., Lennox, A., Spassova, G., & Bragge, P. (2021) · Implementation Science · doi:10.1186/s13012-021-01142-y · Cited by 298
Kang, H. S., & Exworthy, M. (2022) · JMIR mHealth and uHealth · doi:10.2196/35684 · Cited by 259
Deslippe, A. L., Soanes, A., Bouchaud, C. C., Beckenstein, H., Slim, M., Plourde, H., & Cohen, T. R. (2023) · International Journal of Behavioral Nutrition and Physical Activity · doi:10.1186/s12966-023-01424-2 · Cited by 241
“Quality assessment The methodological quality of all papers was assessed using the Critical Appraisal Skills Programme (CASP) checklist for qualitative research [64, 65].”
BackgroundMethod
Metadata
- Authors
- Long, H. A., French, D. P., & Brooks, J. M.
- Affiliations
- University of Manchester (GB)
- Corresponding author
- Long, Hannah A
- Venue
- Research Methods in Medicine & Health Sciences
- Publisher
- SAGE Publications
- Published
- 2020-08-06
- Volume
- 1
- Issue
- 1
- Pages
- 31-42
- ISSN
- 2632-0843 (ISSN-L)
- OpenAlex
- W3047861731
- Cited by
- 1,112
- References
- 49
- Citations by year
- 2026: 188 · 2025: 291 · 2024: 235 · 2023: 218 · 2022: 121 · 2021: 48 · 2020: 7
- FWCI
- 60.06
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Locations
- Research Methods in Medicine & Health Sciences (publishedVersion, cc-by)
- Research Explorer (The University of Manchester) (publishedVersion, other-oa)
- Funders
- National Institute for Health Research Manchester Biomedical Research Centre (IS-BRC-1215-20007)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Education › Qualitative Research Methods and Applications
- Social Sciences › Sociology and Political Science › Delphi Technique in Research
- Keywords
- quality assessment, qualitative evidence synthesis, methodological quality assessment, qualitative research methodology, tool usability, tool modification, user guidance, systematic reviews
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.