Cochrane Qualitative and Implementation Methods Group guidance series-paper 3: methods for assessing methodological limitations, data extraction and synthesis, and confidence in synthesized qualitative findings
Open accessAbstract
The Cochrane Qualitative and Implementation Methods Group develops and publishes guidance on the synthesis of qualitative and mixed-method implementation evidence. Choice of appropriate methodologies, methods, and tools is essential when developing a rigorous protocol and conducting the synthesis. Cochrane authors who conduct qualitative evidence syntheses have thus far used a small number of relatively simple methods to address similarly written questions. Cochrane has invested in methodological work to develop new tools and to encourage the production of exemplar reviews to show the value of more innovative methods that address a wider range of questions. In this paper, in the series, we report updated guidance on the selection of tools to assess methodological limitations in qualitative studies and methods to extract and synthesize qualitative evidence. We recommend application of Grades of Recommendation, Assessment, Development, and Evaluation-Confidence in the Evidence from Qualitative Reviews to assess confidence in qualitative synthesized findings. This guidance aims to support review authors to undertake a qualitative evidence synthesis that is intended to be integrated subsequently with the findings of one or more Cochrane reviews of the effects of similar interventions. The review of intervention effects may be undertaken concurrently with or separate to the qualitative evidence synthesis. We encourage further development through reflection and formal testing.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Included studies were appraised using the EPPI-Centre checklist, which assesses methodological quality across sampling, data collection, analysis, interpretation, and privileging of participant perspectives.
“Included studies were appraised using the EPPI-Centre checklist [25]. This checklist assesses methodological quality across recommended domains: sampling, data collection, analysis, interpretation, and privileging of participant perspectives (see S1 Table) [26, 27].”
From Women’s and girls’ experiences of menstruation in low- and middle-income countries: A systematic review and qualitative metasynthesis (2019) · cites this paper as [26]
This work draws on current best practice guidance.
“This work draws on current best practice guidance [26, 27, 29, 30, 112].”
From Women’s and girls’ experiences of menstruation in low- and middle-income countries: A systematic review and qualitative metasynthesis (2019) · cites this paper as [26]
Cites
Methods for the thematic synthesis of qualitative research in systematic reviews
Thomas, J., & Harden, A. (2008) · BMC Medical Research Methodology · doi:10.1186/1471-2288-8-45 · Cited by 9,312 · see citing works
Self-citationMethods for the synthesis of qualitative research: a critical review
Barnett-Page, E., & Thomas, J. (2009) · BMC Medical Research Methodology · doi:10.1186/1471-2288-9-59 · Cited by 2,087 · see citing works
Self-citationBarone, T., & Eisner, E. W. (2012) · doi:10.4135/9781452230627 · Cited by 1,268 · see citing works
Pluye, P., Gagnon, M.-P., Griffiths, F., & Johnson-Lafleur, J. (2009) · International Journal of Nursing Studies · doi:10.1016/j.ijnurstu.2009.01.009 · Cited by 1,119 · see citing works
Lewin, S. A., Munabi-Babigumira, S. K., Glenton, C., Daniels, K., Bosch‐Capblanch, X., van Wyk, B. E., Odgaard‐Jensen, J., Johansen, M., Aja, G. N., Zwarenstein, M., et al. (2010) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd004015.pub3 · Cited by 986 · see citing works
Self-citationLewin, S., Glenton, C., Munthe-Kaas, H., Carlsen, B., Colvin, C. J., Gülmezoglu, M., Noyes, J., Booth, A., Garside, R., & Rashidian, A. (2015) · PLOS Medicine · doi:10.1371/journal.pmed.1001895 · Cited by 866 · see citing works
Self-citation“Best fit” framework synthesis: refining the method
Carroll, C., Booth, A., Leaviss, J., & Rick, J. (2013) · BMC Medical Research Methodology · doi:10.1186/1471-2288-13-37 · Cited by 649 · see citing works
Evaluating meta-ethnography: systematic analysis and synthesis of qualitative research
Campbell, R., Pound, P., Morgan, M., Daker-White, G., Britten, N., Pill, R., Yardley, L., Pope, C., & Donovan, J. (2011) · Health Technology Assessment · doi:10.3310/hta15430 · Cited by 619 · see citing works
Integrating qualitative research with trials in systematic reviews
Thomas, J., Harden, A., Oakley, A., Oliver, S., Sutcliffe, K., Rees, R., Brunton, G., & Kavanagh, J. (2004) · BMJ · doi:10.1136/bmj.328.7446.1010 · Cited by 515 · cited 3× · see citing works
“What is the likelihood of excluding valuable insights on the basis of quality? Consider if excluding a study will affect assessments of adequacy or coherence of qualitative synthesized findings [16]? Do multiple qualitative studies with methodological concerns report similar consistent findings? Need to report deviant…”
BackgroundMethodInfluentialSelf-citationCarroll, C., Booth, A., & Cooper, K. (2011) · BMC Medical Research Methodology · doi:10.1186/1471-2288-11-29 · Cited by 461 · see citing works
Self-citationGlenton, C., Colvin, C., Carlsen, B., Swartz, A., Lewin, S., Noyes, J., & Rashidian, A. (2013) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd010414.pub2 · Cited by 454 · see citing works
Self-citationBenoot, C., Hannes, K., & Bilsen, J. J. R. (2016) · BMC Medical Research Methodology · doi:10.1186/s12874-016-0114-6 · Cited by 399 · see citing works
Self-citationFlemming, K., Booth, A., Garside, R., Tunçalp, Ö., & Noyes, J. (2019) · BMJ Global Health · doi:10.1136/bmjgh-2018-000882 · Cited by 389 · see citing works
Self-citationUsing logic models to capture complexity in systematic reviews: Logic Models in Systematic Reviews
Anderson, L. M., Petticrew, M., Rehfuess, E., Armstrong, R., Ueffing, E., Baker, P., Francis, D., & Tugwell, P. (2011) · Research Synthesis Methods · doi:10.1002/jrsm.32 · Cited by 372 · see citing works
Malpass, A., Shaw, A., Sharp, D., Walter, F., Feder, G., Ridd, M., & Kessler, D. (2009) · Social Science & Medicine · doi:10.1016/j.socscimed.2008.09.068 · Cited by 368 · see citing works
Hannes, K., & Lockwood, C. S. (2011) · Journal of Advanced Nursing · doi:10.1111/j.1365-2648.2011.05636.x · Cited by 289 · see citing works
Self-citationSystematic Synthesis of Qualitative Research
Saini, M., & Shlonsky, A. (2012) · Oxford University Press eBooks · doi:10.1093/acprof:oso/9780195387216.001.0001 · Cited by 250 · see citing works
Carroll, C., & Booth, A. (2015) · Research Synthesis Methods · doi:10.1002/jrsm.1128 · Cited by 235 · see citing works
Self-citationTugwell, P., Petticrew, M. P., Kristjansson, E., Welch, V. A., Ueffing, E., Waters, E., Bonnefoy, J., Morgan, A., Doohan, E., & Kelly, M. P. (2010) · BMJ · doi:10.1136/bmj.c4739 · Cited by 184 · see citing works
Critical Appraisal of Mixed Methods Studies
Heyvaert, M., Hannes, K., Maes, B., & Onghena, P. (2013) · Journal of Mixed Methods Research · doi:10.1177/1558689813479449 · Cited by 166 · see citing works
Self-citationHarris, J. L., Booth, A., Cargo, M., Hannes, K., Harden, A., Flemming, K., Garside, R., Pantoja, T., Thomas, J., & Noyes, J. (2018) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2017.10.023 · Cited by 165 · see citing works
Self-citationSantiago-Delefosse, M., Gavin, A., Bruchez, C., Roux, P., & Stephen, S. L. (2016) · Social Science & Medicine · doi:10.1016/j.socscimed.2015.11.007 · Cited by 146 · see citing works
Harden, A., Thomas, J., Cargo, M., Harris, J., Tomás Pantoja, C., Flemming, K., Booth, A., Garside, R., Hannes, K., & Noyes, J. (2017) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2017.11.029 · Cited by 146 · see citing works
Self-citationMyth 94: Qualitative Health Researchers Will Agree About Validity
Sparkes, A. C. (2001) · Qualitative Health Research · doi:10.1177/104973230101100409 · Cited by 141 · see citing works
Garside, R. (2013) · Innovation The European Journal of Social Science Research · doi:10.1080/13511610.2013.777270 · Cited by 134 · see citing works
Self-citationFlemming, K., Booth, A., Hannes, K., Cargo, M., & Noyes, J. (2017) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2017.10.022 · Cited by 128 · see citing works
Self-citationKneale, D., Thomas, J., & Harris, K. (2015) · PLoS ONE · doi:10.1371/journal.pone.0142187 · Cited by 126 · see citing works
Self-citationA matter of taste: evaluating the quality of qualitative research
Sandelowski, M. (2014) · Nursing Inquiry · doi:10.1111/nin.12080 · Cited by 112 · see citing works
Aslam, R. W., Hendry, M., Booth, A., Carter, B., Charles, J. M., Craine, N., Edwards, R. T., Noyes, J., Ntambwe, L. I., Pasterfield, D., et al. (2017) · BMC Medicine · doi:10.1186/s12916-017-0904-7 · Cited by 71 · see citing works
Self-citationGarside, R., Britten, N., & Stein, K. (2008) · Journal of Advanced Nursing · doi:10.1111/j.1365-2648.2008.04750.x · Cited by 68 · see citing works
Self-citationCargo, M., Harris, J., Tomás Pantoja, C., Booth, A., Harden, A., Hannes, K., Thomas, J., Flemming, K., Garside, R., & Noyes, J. (2017) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2017.11.028 · Cited by 67 · see citing works
Self-citationSystematic Synthesis of Qualitative Research
Saini, M., & Shlonsky, A. (2012) · Oxford University Press eBooks · doi:10.1093/acprof:oso/9780195387216.003.0005 · Cited by 64 · see citing works
Noyes, J., Hendry, M., Booth, A., Chandler, J., Lewin, S. A., Glenton, C., & Garside, R. (2016) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2015.12.009 · Cited by 58 · see citing works
Self-citationWhitaker, R., Hendry, M., Booth, A., Carter, B., Charles, J. M., Craine, N., Edwards, R., Lyons, M., Noyes, J., Pasterfield, D., et al. (2014) · BMJ Open · doi:10.1136/bmjopen-2013-004733 · Cited by 20 · see citing works
Self-citationTurley, R., Saith, R., Bhan, N., Doyle, J., Jones, K., & Waters, E. (2013) · Journal of Public Health · doi:10.1093/pubmed/fdt008 · Cited by 18 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Long, H. A., French, D. P., & Brooks, J. M. (2020) · Research Methods in Medicine & Health Sciences · doi:10.1177/2632084320947559 · Cited by 1,112 · see citing works
Hennegan, J., Shannon, A. K., Rubli, J., Schwab, K. J., & Melendez-Torres, G. J. (2019) · PLOS Medicine · doi:10.1371/journal.pmed.1002803 · Cited by 577 · see citing works
Improving reporting of meta-ethnography: the eMERGe reporting guidance
France, E. F., Cunningham, M., Ring, N., Uny, I., Duncan, E. A. S., Jepson, R. G., Maxwell, M., Roberts, R. J., Turley, R. L., Booth, A., et al. (2019) · BMC Medical Research Methodology · doi:10.1186/s12874-018-0600-0 · Cited by 469 · see citing works
Self-citationNoyes, J., Booth, A., Moore, G., Flemming, K., Tunçalp, Ö., & Shakibazadeh, E. (2019) · BMJ Global Health · doi:10.1136/bmjgh-2018-000893 · Cited by 497 · see citing works
Self-citationFlemming, K., Booth, A., Garside, R., Tunçalp, Ö., & Noyes, J. (2019) · BMJ Global Health · doi:10.1136/bmjgh-2018-000882 · Cited by 389 · see citing works
Self-citationBreeksema, J. J., Niemeijer, A. R., Krediet, E., Vermetten, E., & Schoevers, R. A. (2020) · CNS Drugs · doi:10.1007/s40263-020-00748-y · Cited by 224 · see citing works
Booth, A., Noyes, J., Flemming, K., Gerhardus, A., Wahlster, P., van der Wilt, G. J., Mozygemba, K., Refolo, P., Sacchini, D., Tummers, M., et al. (2018) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2018.03.003 · Cited by 198 · see citing works
Self-citationExtreme events and gender-based violence: a mixed-methods systematic review
van Daalen, K. R., Kallesøe, S. S., Davey, F., Dada, S., Jung, L., Singh, L., Issa, R., Emilian, C. A., Kuhn, I., Keygnaert, I., et al. (2022) · The Lancet Planetary Health · doi:10.1016/s2542-5196(22)00088-2 · Cited by 199 · see citing works
Buus, N., & Perron, A. (2020) · International Journal of Nursing Studies · doi:10.1016/j.ijnurstu.2019.103452 · Cited by 190 · see citing works
Naidoo, N., Nguyen, V. T., Ravaud, P., Young, B., Amiel, P., Schanté, D., Clarke, M., & Boutron, I. (2020) · BMC Medicine · doi:10.1186/s12916-019-1476-5 · Cited by 144 · see citing works
Benefits and challenges of cancer peer support groups: A systematic review of qualitative studies
Jablotschkin, M., Binkowski, L., Markovits Hoopii, R., & Weis, J. (2022) · European Journal of Cancer Care · doi:10.1111/ecc.13700 · Cited by 88 · see citing works
Physical workplace adjustments to support neurodivergent workers: A systematic review
Weber, C., Krieger, B., Häne, E., Yarker, J., & McDowall, A. (2024) · Applied Psychology · doi:10.1111/apps.12431 · Cited by 86 · see citing works
Young, A. E., Brookes, S. T., Avery, K. N. L., Davies, A., Metcalfe, C., & Blazeby, J. M. (2019) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2019.06.016 · Cited by 77 · see citing works
Hennelly, N., Cooney, A., Houghton, C., & O’Shea, E. (2021) · The Gerontologist · doi:10.1093/geront/gnz159 · Cited by 61 · see citing works
Niño de Guzmán Quispe, E., Martínez García, L., Orrego Villagrán, C., Heijmans, M., Sunol, R., Fraile-Navarro, D., Pérez-Bracchiglione, J., Ninov, L., Salas-Gama, K., Viteri García, A., et al. (2021) · The Patient - Patient-Centered Outcomes Research · doi:10.1007/s40271-021-00514-2 · Cited by 54 · see citing works
Juel, A., Berring, L. L., Hybholt, L., Erlangsen, A., Larsen, E. R., & Buus, N. (2021) · International Journal of Nursing Studies · doi:10.1016/j.ijnurstu.2020.103793 · Cited by 20 · see citing works
Li, J., Ma, L., Sun, H., Li, M., Cao, Y., Peng, Y., & Xu, J. (2025) · European Archives of Psychiatry and Clinical Neuroscience · doi:10.1007/s00406-024-01920-x · Cited by 5 · see citing works
- Authors
- Noyes, J., Booth, A., Flemming, K., Garside, R., Harden, A., Lewin, S., Pantoja, T., Hannes, K., Cargo, M., & Thomas, J.
- Affiliations
- Bangor University (GB)
- University of Sheffield (GB)
- University of York (GB)
- University of Exeter (GB)
- University of East London (GB)
- South African Medical Research Council (ZA)
- KU Leuven (BE)
- University of South Australia (AU)
- University College London (GB)
- Corresponding author
- Noyes, Jane
- ORCID
- Venue
- Journal of Clinical Epidemiology
- Publisher
- Elsevier BV
- Published
- 2018-05
- Volume
- 97
- Pages
- 49-58
- ISSN
- 0895-4356 (ISSN-L)
- OpenAlex
- W2775212908
- Cited by
- 480
- References
- 35
- Citations by year
- 2026: 54 · 2025: 69 · 2024: 67 · 2023: 72 · 2022: 60 · 2021: 45 · 2020: 49 · 2019: 51 · 2018: 10 · 2017: 3
- FWCI
- 27.65
- Type
- journal article
- Language
- en
- Open access
- green
- doi.org
- Licence
- cc-by-nc-nd
- www.elsevier.com/tdm/userlicense/1.0/ (tdm)
- Locations
- Journal of Clinical Epidemiology (publishedVersion)
- PubMed (publishedVersion)
- UEL Research Repository (University of East London) (acceptedVersion)
- Adelaide Research & Scholarship (AR&S) (University of Adelaide) (submittedVersion)
- UCL Discovery (University College London) (submittedVersion)
- UEL Research Repository (University of East London) (submittedVersion, cc-by-nc-nd)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Health Sciences › Health Professions › General Health Professions › Health Policy Implementation Science
- Social Sciences › Education › Qualitative Research Methods and Applications
- Keywords
- qualitative evidence synthesis, data extraction
- MeSH terms
- Data Accuracy, Systematic Reviews as Topic, Data Analysis, Decision Making, Humans, Evidence-Based Medicine, Biomedical Research, Qualitative Research
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review