Rationale and Standards for the Systematic Review of Qualitative Literature in Health Services Research
Abstract
Despite growing recognition of the need for qualitative methods in health services research, there have been few attempts to define quality standards for assessing the results. This article acknowledges the desirability of a plurality of standards. However, it is argued that three interrelated criteria can be identified as the foundation of good qualitative health research: interpretation of subjective meaning, description of social context, and attention to lay knowledge. These criteria can be examined in relation to different dimensions of any research report, including theoretical basis, sampling strategy, scope of data collection, description of data collected, and concern with generalizability or typicality. But if the concern is with the appropriateness of care and with understanding the factors that shape lay and clinical behavior, then these criteria must form the basis of a hierarchy of qualitative research evidence.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Summarized data kept within the wider context of each case encourages thick description that pays attention to complex layers of meaning and understanding.
“The summarized data is kept within the wider context of each case, thereby encouraging thick description that pays attention to complex layers of meaning and understanding [38].”
From Using the framework method for the analysis of qualitative data in multi-disciplinary health research (2013) · cites this paper as [38]
Checklists have contributed to the wider acceptance of qualitative research methods and to convincing potential sceptics of its thoroughness.
“Checklists have played an important role in conferring respectability on qualitative research and in convincing potential sceptics of its thoroughness.1–3”
From Checklists for improving rigour in qualitative research: a case of the tail wagging the dog? (2001) · cites this paper as [1]
Qualitative research is usually carried out from a relativist perspective, which acknowledges the existence of multiple views of equal validity.
“Qualitative research, however, is usually carried out from a relativist perspective, which acknowledges the existence of multiple views of equal validity.1”
From Checklists for improving rigour in qualitative research: a case of the tail wagging the dog? (2001) · cites this paper as [1]
The greatest potential of multiple coding lies in its capacity to furnish alternative interpretations and thereby to act as the "devil's advocate" in alerting researchers to all potentially competing explanations.
“The greatest potential of multiple coding lies in its capacity to furnish alternative interpretations and thereby to act as the “devil's advocate” implied in many of the checklists1–3 in alerting researchers to all potentially competing explanations.”
From Checklists for improving rigour in qualitative research: a case of the tail wagging the dog? (2001) · cites this paper as [1]
Cites
Qualitative methods for assessing health care.
Fitzpatrick, R., & Boulton, M. (1994) · Quality in Health Care · doi:10.1136/qshc.3.2.107 · Cited by 298 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Tong, A., Sainsbury, P., & Craig, J. (2007) · International Journal for Quality in Health Care · doi:10.1093/intqhc/mzm042 · Cited by 42,509 · see citing works
Tranfield, D., Denyer, D., & Smart, P. (2003) · British Journal of Management · doi:10.1111/1467-8551.00375 · Cited by 15,386 · see citing works
Standards for Reporting Qualitative Research: A Synthesis of Recommendations
O’Brien, B. C., Harris, I. B., Beckman, T. J., Reed, D. A., & Cook, D. A. (2014) · Academic Medicine · doi:10.1097/acm.0000000000000388 · Cited by 12,211 · see citing works
Gale, N. K., Heath, G., Cameron, E., Rashid, S., & Redwood, S. (2013) · BMC Medical Research Methodology · doi:10.1186/1471-2288-13-117 · Cited by 10,865 · see citing works
Verification strategies for establishing reliability and validity in qualitative research
Morse, J. M., Barrett, M., Mayan, M., Olson, K., & Spiers, J. (2002) · International Journal of Qualitative Methods · doi:10.1177/160940690200100202 · Cited by 4,978 · see citing works
Intercoder Reliability in Qualitative Research: Debates and Practical Guidelines
O’Connor, C., & Joffe, H. (2020) · International Journal of Qualitative Methods · doi:10.1177/1609406919899220 · Cited by 3,871 · see citing works
“Qualitative researchers have proposed alternative quality criteria that substitute the typical standards used in evaluating quantitative research (Bauer et al., 2000; Popay et al., 1998; Seale & Silverman, 1997; Yardley, 2000).”
BackgroundEnhancing transparency in reporting the synthesis of qualitative research: ENTREQ
Tong, A., Flemming, K., McInnes, E., Oliver, S., & Craig, J. (2012) · BMC Medical Research Methodology · doi:10.1186/1471-2288-12-181 · Cited by 3,569 · see citing works
“Some reviewers have developed their own appraisal framework selecting items from existing criteria [25,39-41], augmented with additional criteria they deemed were specifically relevant to the research topic.”
BackgroundChecklists for improving rigour in qualitative research: a case of the tail wagging the dog?
Barbour, R. S. (2001) · BMJ · doi:10.1136/bmj.322.7294.1115 · Cited by 2,613 · see citing works
Understanding and Evaluating Qualitative Research
Fossey, E., Harvey, C., Mcdermott, F., & Davidson, L. (2002) · Australian & New Zealand Journal of Psychiatry · doi:10.1046/j.1440-1614.2002.01100.x · Cited by 2,311 · see citing works
Validity in qualitative research
Whittemore, R., Chase, S. K., & Mandle, C. L. (2001) · Qualitative Health Research · doi:10.1177/104973201129119299 · Cited by 2,144 · see citing works
Standard Quality Assessment Criteria for Evaluating Primary Research Papers from a Variety of Fields
Kmet, L. M., Cook, L. S., & Lee, R. C. (2004) · University of Alberta Library · doi:10.7939/r37m04f16 · Cited by 1,682 · see citing works
Validity, reliability, and generalizability in qualitative research
Leung, L. (2015) · Journal of Family Medicine and Primary Care · doi:10.4103/2249-4863.161306 · Cited by 1,404 · see citing works
Meta‐synthesis method for qualitative research: a literature review
Walsh, D., & Downe, S. (2005) · Journal of Advanced Nursing · doi:10.1111/j.1365-2648.2005.03380.x · Cited by 1,246 · see citing works
Appraising the Evidence: Reviewing Disparate Data Systematically
Hawker, S., Payne, S., Kerr, C., Hardey, M., & Powell, J. (2002) · Qualitative Health Research · doi:10.1177/1049732302238251 · Cited by 1,216 · see citing works
Pluye, P., Gagnon, M.-P., Griffiths, F., & Johnson-Lafleur, J. (2009) · International Journal of Nursing Studies · doi:10.1016/j.ijnurstu.2009.01.009 · Cited by 1,119 · see citing works
Criteria for evaluating evidence on public health interventions
Rychetnik, L., Frommer, M., Hawe, P., & Shiell, A. (2002) · Journal of Epidemiology and Community Health · doi:10.1136/jech.56.2.119 · Cited by 1,042 · see citing works
Campbell, R., Pound, P., Pope, C., Britten, N., Pill, R., Morgan, M., & Donovan, J. (2003) · Social Science & Medicine · doi:10.1016/s0277-9536(02)00064-3 · Cited by 921 · see citing works
Critically appraising qualitative research
Kuper, A., Lingard, L., & Levinson, W. (2008) · BMJ · doi:10.1136/bmj.a1035 · Cited by 784 · see citing works
O’Connor, S., Hanlon, P., O’Donnell, C. A., Garcia, S., Glanville, J., & Mair, F. S. (2016) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-016-0359-3 · Cited by 636 · see citing works
The problem of appraising qualitative research
Dixon-Woods, M. (2004) · Quality and Safety in Health Care · doi:10.1136/qshc.2003.008714 · Cited by 653 · see citing works
“Best fit” framework synthesis: refining the method
Carroll, C., Booth, A., Leaviss, J., & Rick, J. (2013) · BMC Medical Research Methodology · doi:10.1186/1471-2288-13-37 · Cited by 649 · see citing works
Evaluating meta-ethnography: systematic analysis and synthesis of qualitative research
Campbell, R., Pound, P., Morgan, M., Daker-White, G., Britten, N., Pill, R., Yardley, L., Pope, C., & Donovan, J. (2011) · Health Technology Assessment · doi:10.3310/hta15430 · Cited by 619 · see citing works
Criteria for the systematic review of health promotion and public health interventions
Jackson, N., & Waters, E. (2005) · Health Promotion International · doi:10.1093/heapro/dai022 · Cited by 572 · see citing works
Systematic Reviews from Astronomy to Zoology: Myths and Misconceptions
Petticrew, M. (2001) · BMJ · doi:10.1136/bmj.322.7278.98 · Cited by 550 · see citing works
Evaluation of qualitative research
Horsburgh, D. (2003) · Journal of Clinical Nursing · doi:10.1046/j.1365-2702.2003.00683.x · Cited by 459 · see citing works
Enhancing the Rigor of Grounded Theory: Incorporating Reflexivity and Relationality
Hall, W. A., & Callery, P. (2001) · Qualitative Health Research · doi:10.1177/104973201129119082 · Cited by 430 · see citing works
Evidence-based Policy: In Search of a Method
Pawson, R. (2002) · Evaluation · doi:10.1177/1358902002008002512 · Cited by 403 · see citing works
Mills, E., Jadad, A. R., Ross, C., & Wilson, K. (2005) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2005.09.002 · Cited by 374 · see citing works
Aitken, M., de St. Jorre, J., Pagliari, C., Jepson, R., & Cunningham-Burley, S. (2016) · BMC Medical Ethics · doi:10.1186/s12910-016-0153-x · Cited by 346 · see citing works
Harden, A., Garcia, J., Oliver, S., Rees, R., Shepherd, J., Brunton, G., & Oakley, A. (2004) · Journal of Epidemiology and Community Health · doi:10.1136/jech.2003.014829 · Cited by 328 · see citing works
Toward an agenda for evaluation of qualitative research
Stige, B., Malterud, K., & Midtgarden, T. (2009) · Qualitative Health Research · doi:10.1177/1049732309348501 · Cited by 327 · see citing works
Balancing the evidence: incorporating the synthesis of qualitative data into systematic reviews
Pearson, A. (2004) · JBI Reports · doi:10.1111/j.1479-6988.2004.00008.x · Cited by 305 · see citing works
The positive aspects of caregiving in dementia: A critical review of the qualitative literature
Lloyd, J., Patterson, T., & Muers, J. (2016) · Dementia · doi:10.1177/1471301214564792 · Cited by 300 · see citing works
Wong, G., Greenhalgh, T., Westhorp, G., & Pawson, R. (2014) · Health Services and Delivery Research · doi:10.3310/hsdr02300 · Cited by 276 · see citing works
Carroll, C., Booth, A., & Lloyd-Jones, M. (2012) · Qualitative Health Research · doi:10.1177/1049732312452937 · Cited by 248 · see citing works
Oliver, S., Harden, A., Rees, R., Shepherd, J., Brunton, G., Garcia, J., & Oakley, A. (2005) · Evaluation · doi:10.1177/1356389005059383 · Cited by 221 · see citing works
Buus, N., & Perron, A. (2020) · International Journal of Nursing Studies · doi:10.1016/j.ijnurstu.2019.103452 · Cited by 190 · see citing works
A methodological systematic review of what’s wrong with meta-ethnography reporting
France, E. F., Ring, N., Thomas, R., Noyes, J., Maxwell, M., & Jepson, R. (2014) · BMC Medical Research Methodology · doi:10.1186/1471-2288-14-119 · Cited by 183 · see citing works
Recommendations to advance digital health equity: A systematic review of qualitative studies
Wilson, S., Tolley, C., Mc Ardle, R., Lawson, L., Beswick, E., Hassan, N., Slight, R., & Slight, S. (2024) · npj Digital Medicine · doi:10.1038/s41746-024-01177-7 · Cited by 139 · see citing works
Caron, C. D., Ducharme, F., & Griffith, J. (2006) · Canadian Journal on Aging / La Revue canadienne du vieillissement · doi:10.1353/cja.2006.0033 · Cited by 122 · see citing works
Fletcher, B. R., Hinton, L., Hartmann-Boyce, J., Roberts, N. W., Bobrovitz, N., & McManus, R. J. (2016) · Patient Education and Counseling · doi:10.1016/j.pec.2015.08.026 · Cited by 121 · see citing works
“It has been argued that three criteria are the foundation of good qualitative reviews: interpretation of subjective meaning; description of social context; and attention to lay knowledge [13].”
BackgroundThe newfound credibility of qualitative research? tales of technical essentialism and co-option
Barbour, R. S. (2003) · Qualitative Health Research · doi:10.1177/1049732303253331 · Cited by 108 · see citing works
Roen, K., Arai, L., Roberts, H., & Popay, J. (2006) · Social Science & Medicine · doi:10.1016/j.socscimed.2006.02.013 · Cited by 106 · see citing works
Self-citationSchmid, M., Brianza, E., Mok, S. Y., & Petko, D. (2024) · Computers & Education · doi:10.1016/j.compedu.2024.105024 · Cited by 87 · see citing works
Parenting support in the context of poverty: a meta-synthesis of the qualitative evidence
Attree, P. (2005) · Health and Social Care in the Community · doi:10.1111/j.1365-2524.2005.00562.x · Cited by 74 · see citing works
South, J., Bagnall, A.-M., Hulme, C., Woodall, J., Longo, R., Dixey, R., Kinsella, K., Raine, G., Vinall-Collier, K., & Wright, J. (2014) · Health Services and Delivery Research · doi:10.3310/hsdr02350 · Cited by 63 · see citing works
Stevenson, M., Scope, A., Sutcliffe, P., Booth, A., Slade, P., Parry, G., Saxon, D., & Kalthenthaler, E. (2010) · Health Technology Assessment · doi:10.3310/hta14440 · Cited by 57 · see citing works
Active Patient Participation in the Development of an Online Intervention
van Bruinessen, I. R., van Weel-Baumgarten, E. M., Snippe, H. W., Gouw, H., Zijlstra, J. M., & van Dulmen, S. (2014) · JMIR Research Protocols · doi:10.2196/resprot.3695 · Cited by 43 · see citing works
Cena, E., Brooks, J., Day, W., Goodman, S., Rousaki, A., Ruby-Granger, V., & Seymour-Smith, S. (2024) · International Journal of Qualitative Methods · doi:10.1177/16094069241282843 · Cited by 49 · see citing works
“Some researchers have attempted to show how quantitative criteria can be applied to qualitative research by providing checklists that can operate as universal standards in assessing qualitative work (Cesario et al., 2002; O ’ Brien et al., 2014; Popay & Rogers, 1998).”
ASTHANA, S., & HALLIDAY, J. (2006) · The Milbank Quarterly · doi:10.1111/j.1468-0009.2006.00459.x · Cited by 45 · see citing works
Adolescents’ and Young Adults’ Beliefs about Mental Health Services and Care: A Systematic Review
Goodwin, J., Savage, E., & Horgan, A. (2016) · Archives of Psychiatric Nursing · doi:10.1016/j.apnu.2016.04.004 · Cited by 43 · see citing works
Living with nuclear energy: A systematic review of the psychological consequences of nuclear power
Edwards, M. W., Schweitzer, R. D., Shakespeare-Finch, J., Byrne, A., & Gordon-King, K. (2019) · Energy Research & Social Science · doi:10.1016/j.erss.2018.08.016 · Cited by 41 · see citing works
Assessing the cultural competence of oral health research conducted with migrant children
Riggs, E., Gussy, M., Gibbs, L., van Gemert, C., Waters, E., Priest, N., Watt, R., Renzaho, A. M. N., & Kilpatrick, N. (2014) · Community Dentistry and Oral Epidemiology · doi:10.1111/cdoe.12058 · Cited by 36 · see citing works
Systematic reviews of qualitative evidence: what are the experiences of women with endometriosis?
Denny, E., & Khan, K. S. (2006) · Journal of Obstetrics and Gynaecology · doi:10.1080/01443610600797301 · Cited by 35 · see citing works
Adaptive reinventing: implicit bias and the co-construction of social change
Sukhera, J., Milne, A., Teunissen, P. W., Lingard, L., & Watling, C. (2018) · Advances in Health Sciences Education · doi:10.1007/s10459-018-9816-3 · Cited by 36 · see citing works
Cassidy, S., Turnbull, S., & Gumley, A. (2014) · The Arts in Psychotherapy · doi:10.1016/j.aip.2014.07.003 · Cited by 33 · see citing works
Han, B., Nawaz, S., Buchanan, G., & McKay, D. (2025) · International Journal of Artificial Intelligence in Education · doi:10.1007/s40593-024-00456-4 · Cited by 32 · see citing works
Moving beyond floccinaucinihilipilification: Enhancing the utility of systematic reviews
Popay, J. (2005) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2005.08.004 · Cited by 29 · see citing works
Self-citationO’Callaghan, C., Dwyer, J., & Schofield, P. (2024) · Qualitative Research in Psychology · doi:10.1080/14780887.2024.2347580 · Cited by 26 · see citing works
(2006) · Systematic Reviews in the Social Sciences · doi:10.1002/9780470754887.ch6 · Cited by 12 · see citing works
- Authors
- Popay, J., Rogers, A., & Williams, G.
- Affiliations
- University of Salford (GB)
- University of Manchester (GB)
- Venue
- Qualitative Health Research
- Publisher
- SAGE Publications
- Published
- 1998-05
- Volume
- 8
- Issue
- 3
- Pages
- 341-351
- ISSN
- 1049-7323 (ISSN-L)
- 1552-7557
- OpenAlex
- W2106339945
- Cited by
- 987
- References
- 1
- Citations by year
- 2026: 11 · 2025: 24 · 2024: 39 · 2023: 34 · 2022: 28 · 2021: 31 · 2020: 35 · 2019: 37 · 2018: 42 · 2017: 38 · 2016: 50 · 2015: 55 · 2014: 51 · 2013: 62 · 2012: 51
- FWCI
- 20.69
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Qualitative Health Research (publishedVersion)
- PubMed (publishedVersion)
- Lancaster EPrints (Lancaster University) (submittedVersion)
- Research Explorer (The University of Manchester) (submittedVersion)
- Topics
- Social Sciences › Education › Qualitative Research Methods and Applications
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Health Sciences › Health Professions › General Health Professions › Health Policy Implementation Science
- Keywords
- health services research, social environment, lay knowledge, sampling strategy, generalizability
- MeSH terms
- Systematic Reviews as Topic, Evaluation Studies as Topic, Health Services Research, Humans, Quality of Health Care, Research Design
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
