The DOI matches its Crossref record; 63 of the 63 references it cites were found in a registry or an index.
Synthesising quantitative and qualitative evidence to inform guidelines on complex interventions: clarifying the purposes, designs and outlining some methods
Open accessAbstract
Guideline developers are increasingly dealing with more difficult decisions concerning whether to recommend complex interventions in complex and highly variable health systems. There is greater recognition that both quantitative and qualitative evidence can be combined in a mixed-method synthesis and that this can be helpful in understanding how complexity impacts on interventions in specific contexts. This paper aims to clarify the different purposes, review designs, questions, synthesis methods and opportunities to combine quantitative and qualitative evidence to explore the complexity of complex interventions and health systems. Three case studies of guidelines developed by WHO, which incorporated quantitative and qualitative evidence, are used to illustrate possible uses of mixed-method reviews and evidence. Additional examples of methods that can be used or may have potential for use in a guideline process are outlined. Consideration is given to the opportunities for potential integration of quantitative and qualitative evidence at different stages of the review and guideline process. Encouragement is given to guideline commissioners and developers and review authors to consider including quantitative and qualitative evidence. Recommendations are made concerning the future development of methods to better address questions in systematic reviews and guidelines that adopt a complexity perspective.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“the first author read all the charted data (including study aims, methods, and samples) before coding each study’s findings to preserve its original context and ensure that its findings could be fully understood without misinterpretation [39].”
From Experiences of Using Digital Mindfulness-Based Interventions: Rapid Scoping Review and Thematic Synthesis (2023) · cites this paper as [39]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Tong, A., Sainsbury, P., & Craig, J. (2007) · International Journal for Quality in Health Care · doi:10.1093/intqhc/mzm042 · Cited by 42,509
The Cochrane Collaboration's tool for assessing risk of bias in randomised trials
Higgins, J. P. T., Altman, D. G., Gotzsche, P. C., Juni, P., Moher, D., Oxman, A. D., Savovic, J., Schulz, K. F., Weeks, L., Sterne, J. A. C., et al. (2011) · BMJ · doi:10.1136/bmj.d5928 · Cited by 33,817
Cochrane Handbook for Systematic Reviews of Interventions: Cochrane Book Series
(2008) · doi:10.1002/9780470712184 · Cited by 25,048
Dermyshi, E., Wang, Y., Yan, C., Hong, W., Qiu, G., Gong, X., & Zhang, T. (2017) · Neonatology · doi:10.1159/000454668 · Cited by 24,586
Designing and Conducting Mixed Methods Research
Robinson, P. M. (2007) · Australian and New Zealand Journal of Public Health · doi:10.1111/j.1753-6405.2007.00096.x · Cited by 23,362
Methods for the thematic synthesis of qualitative research in systematic reviews
Thomas, J., & Harden, A. (2008) · BMC Medical Research Methodology · doi:10.1186/1471-2288-8-45 · Cited by 9,312
(2006) · Cited by 7,081
SAGE Handbook of Mixed Methods in Social & Behavioral Research
Tashakkori, A., & Teddlie, C. (2010) · doi:10.4135/9781506335193 · Cited by 6,215
Process evaluation of complex interventions: Medical Research Council guidance
Moore, G. F., Audrey, S., Barker, M., Bond, L., Bonell, C., Hardeman, W., Moore, L., O'Cathain, A., Tinati, T., Wight, D., et al. (2015) · BMJ · doi:10.1136/bmj.h1258 · Cited by 6,204
Self-citationNoblit, G., & Hare, R. (1988) · doi:10.4135/9781412985000 · Cited by 2,723
Meta-Ethnography: Synthesizing Qualitative Studies
Noblit, G. W., & Hare, R. D. (1988) · Medical Entomology and Zoology · Cited by 2,191
Pluye, P., & Hong, Q. N. (2014) · Annual Review of Public Health · doi:10.1146/annurev-publhealth-032013-182440 · Cited by 1,363
Hong, Q. N., Gonzalez‐Reyes, A., & Pluye, P. (2018) · Journal of Evaluation in Clinical Practice · doi:10.1111/jep.12884 · Cited by 1,107
Evidence-Based Public Health: Moving Beyond Randomized Trials
Victora, C. G., Habicht, J.-P., & Bryce, J. (2004) · American Journal of Public Health · doi:10.2105/ajph.94.3.400 · Cited by 1,056
Storylines of research in diffusion of innovation: a meta-narrative approach to systematic review
Greenhalgh, T., Robert, G., Macfarlane, F., Bate, P., Kyriakidou, O., & Peacock, R. (2005) · Social Science & Medicine · doi:10.1016/j.socscimed.2004.12.001 · Cited by 913
Lewin, S., Glenton, C., Munthe-Kaas, H., Carlsen, B., Colvin, C. J., Gülmezoglu, M., Noyes, J., Booth, A., Garside, R., & Rashidian, A. (2015) · PLOS Medicine · doi:10.1371/journal.pmed.1001895 · Cited by 866
Self-citationMixing Methods: qualitative and quantitative research
(2017) · Routledge eBooks · doi:10.4324/9781315248813 · Cited by 787
Hong, Q. N., Pluye, P., Bujold, M., & Wassef, M. (2017) · Systematic Reviews · doi:10.1186/s13643-017-0454-2 · Cited by 754
Guidance on the conduct of narrative synthesis in sytematic reviews
Popay, J., Roberts, H. M., Sowden, A. J., Petticrew, M. P., Arai, L., Rodgers, M., & Britten, N. (2006) · Cited by 690
“Best fit” framework synthesis: refining the method
Carroll, C., Booth, A., Leaviss, J., & Rick, J. (2013) · BMC Medical Research Methodology · doi:10.1186/1471-2288-13-37 · Cited by 649
Rodgers, M., Sowden, A., Petticrew, M., Arai, L., Roberts, H., Britten, N., & Popay, J. (2009) · Evaluation · doi:10.1177/1356389008097871 · Cited by 561
A Systematic Review of Theory-Driven Evaluation Practice From 1990 to 2009
Coryn, C. L. S., Noakes, L. A., Westine, C. D., & Schröter, D. C. (2011) · American Journal of Evaluation · doi:10.1177/1098214010389321 · Cited by 531
Integrating qualitative research with trials in systematic reviews
Thomas, J., Harden, A., Oakley, A., Oliver, S., Sutcliffe, K., Rees, R., Brunton, G., & Kavanagh, J. (2004) · BMJ · doi:10.1136/bmj.328.7446.1010 · Cited by 515
Southern, D. A., Norris, C. M., Quan, H., Shrive, F. M., Galbraith, P. D., Humphries, K. H., Gao, M., Knudtson, M. L., Ghali, W. A., & the APPROACH Investigators (2008) · BMC Medical Research Methodology · doi:10.1186/1471-2288-8-1 · Cited by 503
Using framework-based synthesis for conducting reviews of qualitative studies
Dixon‐Woods, M. (2011) · BMC Medicine · doi:10.1186/1741-7015-9-39 · Cited by 496
(2006) · PubMed · Cited by 485
Noyes, J., Booth, A., Flemming, K., Garside, R., Harden, A., Lewin, S., Pantoja, T., Hannes, K., Cargo, M., & Thomas, J. (2018) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2017.06.020 · Cited by 480
Self-citationO’Cathain, A., Hoddinott, P., Lewin, S., Thomas, K. J., Young, B., Adamson, J., Jansen, Y. J., Mills, N., Moore, G., & Donovan, J. L. (2015) · Pilot and Feasibility Studies · doi:10.1186/s40814-015-0026-y · Cited by 458
Self-citationAtun, R., de Jongh, T., Secci, F., Ohiri, K., & Adeyi, O. (2010) · Health Policy and Planning · doi:10.1093/heapol/czp055 · Cited by 427
Flemming, K., Booth, A., Garside, R., Tunçalp, Ö., & Noyes, J. (2019) · BMJ Global Health · doi:10.1136/bmjgh-2018-000882 · Cited by 389
Self-citationUsing logic models to capture complexity in systematic reviews: Logic Models in Systematic Reviews
Anderson, L. M., Petticrew, M., Rehfuess, E., Armstrong, R., Ueffing, E., Baker, P., Francis, D., & Tugwell, P. (2011) · Research Synthesis Methods · doi:10.1002/jrsm.32 · Cited by 372
Researching public health: Behind the qualitative-quantitative methodological debate
Baum, F. (1995) · Social Science & Medicine · doi:10.1016/0277-9536(94)e0103-y · Cited by 335
Petticrew, M. P., Rehfuess, E. A., Noyes, J., Higgins, J. P. T., Mayhew, A., Tomás Pantoja, C., Shemilt, I., & Sowden, A. J. (2013) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2013.06.005 · Cited by 328
Self-citationSynthesising quantitative evidence in systematic reviews of complex health interventions
Higgins, J. P. T., López-López, J. A., Becker, B. J., Davies, S. R., Dawson, S., Grimshaw, J. M., McGuinness, L. A., Moore, T. H. M., Rehfuess, E. A., Thomas, J., et al. (2019) · BMJ Global Health · doi:10.1136/bmjgh-2018-000858 · Cited by 322
Mixed Methods Research for Nursing and the Health Sciences
(2009) · Wiley eBooks · doi:10.1002/9781444316490 · Cited by 315
Fletcher, A., Jamal, F., Moore, G., Evans, R. E., Murphy, S., & Bonell, C. (2016) · Evaluation · doi:10.1177/1356389016652743 · Cited by 294
Self-citationMapping the Mixed Methods–Mixed Research Synthesis Terrain
Sandelowski, M., Voils, C. I., Leeman, J., & Crandell, J. (2011) · Journal of Mixed Methods Research · doi:10.1177/1558689811427913 · Cited by 290
Deconstructing the qualitative-quantitative divide 1
Hammersley, M. (2017) · Routledge eBooks · doi:10.4324/9781315248813-2 · Cited by 275
Formulating questions to explore complex interventions within qualitative evidence synthesis
Booth, A., Noyes, J., Flemming, K., Moore, G., Tunçalp, Ö., & Shakibazadeh, E. (2019) · BMJ Global Health · doi:10.1136/bmjgh-2018-001107 · Cited by 258
Self-citationPetticrew, M. P., Knai, C., Thomas, J., Rehfuess, E. A., Noyes, J., Gerhardus, A., Grimshaw, J., Rutter, H., & McGill, E. (2019) · BMJ Global Health · doi:10.1136/bmjgh-2018-000899 · Cited by 212
Self-citationRehfuess, E. A., Stratil, J. M., Scheel, I. B., Portela, A. G., Norris, S. L., & Baltussen, R. M. P. M. (2019) · BMJ Global Health · doi:10.1136/bmjgh-2018-000844 · Cited by 209
Raine, R., Fitzpatrick, R. M., Barratt, H., Bevan, G., Black, N. A., Boaden, R., Bower, P., Campbell, M., Denis, J., Devers, K. J., et al. (2016) · Health Services and Delivery Research · doi:10.3310/hsdr04160 · Cited by 197
Self-citationFood sovereignty, food security and health equity: a meta-narrative mapping exercise
Weiler, A. M., Hergesheimer, C., Brisbois, B., Wittman, H. K., Yassi, A., & Spiegel, J. (2014) · Health Policy and Planning · doi:10.1093/heapol/czu109 · Cited by 194
Using Mixed Methods Research Synthesis for Literature Reviews
Heyvaert, M., Hannes, K., & Onghena, P. (2017) · doi:10.4135/9781506333243 · Cited by 164
Roberts, K. A., Dixon‐Woods, M., Fitzpatrick, R. M., Abrams, K. R., & Jones, D. R. (2002) · The Lancet · doi:10.1016/s0140-6736(02)11560-1 · Cited by 159
O’Cathain, A., Thomas, K. J., Drabble, S. J., Rudolph, A., Goode, J., & Hewison, J. (2014) · Health Technology Assessment · doi:10.3310/hta18380 · Cited by 159
Harden, A., Thomas, J., Cargo, M., Harris, J., Tomás Pantoja, C., Flemming, K., Booth, A., Garside, R., Hannes, K., & Noyes, J. (2017) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2017.11.029 · Cited by 157
Self-citationMixed Methods and Systematic Reviews: Examples and Emerging Issues
Harden, A., & Thomas, J. (2010) · Sage eBooks · doi:10.4135/9781506335193.n29 · Cited by 150
Synthesis of quantitative and qualitative research: an example using Critical Interpretive Synthesis
Flemming, K. (2009) · Journal of Advanced Nursing · doi:10.1111/j.1365-2648.2009.05173.x · Cited by 146
Self-citationFlemming, K., Booth, A., Hannes, K., Cargo, M., & Noyes, J. (2017) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2017.10.022 · Cited by 133
Self-citationTaking account of context in systematic reviews and guidelines considering a complexity perspective
Booth, A., Moore, G. F., Flemming, K., Garside, R., Rollins, N. C., Tunçalp, Ö., & Noyes, J. (2019) · BMJ Global Health · doi:10.1136/bmjgh-2018-000840 · Cited by 130
Self-citationKneale, D., Thomas, J., & Harris, K. (2015) · PLoS ONE · doi:10.1371/journal.pone.0142187 · Cited by 126
Mixed Methods Intervention Trials
Creswell, J. W., Fetters, M. D., Plano Clark, V. L., & Morales, A. (2009) · Mixed Methods Research for Nursing and the Health Sciences · doi:10.1002/9781444316490.ch9 · Cited by 99
Providing effective and preferred care closer to home: a realist review of intermediate care
Pearson, M., Hunt, H. A., Cooper, C., Shepperd, S., Pawson, R., & Anderson, R. (2015) · Health & Social Care in the Community · doi:10.1111/hsc.12183 · Cited by 81
Tricco, A. C., Antony, J., Soobiah, C., Kastner, M., MacDonald, H. R., Cogo, E., Lillie, E., Tran, J., & Straus, S. E. (2016) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2015.12.011 · Cited by 80
Montgomery, P., Movsisyan, A., Grant, S., Macdonald, G. M., & Rehfuess, E. A. (2019) · BMJ Global Health · doi:10.1136/bmjgh-2018-000848 · Cited by 72
Chandler, J., Rycroft‐Malone, J., Hawkes, C., & Noyes, J. (2016) · Journal of Advanced Nursing · doi:10.1111/jan.12815 · Cited by 70
Self-citationCargo, M., Harris, J., Tomás Pantoja, C., Booth, A., Harden, A., Hannes, K., Thomas, J., Flemming, K., Garside, R., & Noyes, J. (2017) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2017.11.028 · Cited by 67
Self-citationNoyes, J., Hendry, M., Booth, A., Chandler, J., Lewin, S. A., Glenton, C., & Garside, R. (2016) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2015.12.009 · Cited by 58
Self-citationKelly, M. P., Noyes, J., Kane, R. L., Chang, C., Uhl, S., Robinson, K. A., Springs, S., Butler, M. E., & Guise, J.-M. (2017) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2017.06.012 · Cited by 49
Self-citationMixed Methods Research Prevalence Studies
Molina-Azorín, J. F., & Fetters, M. D. (2016) · Journal of Mixed Methods Research · doi:10.1177/1558689816636707 · Cited by 47
Weir, M., Grimshaw, J., Mayhew, A., & Fergusson, D. A. (2012) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2011.12.012 · Cited by 38
Turley, R., Saith, R., Bhan, N., Doyle, J., Jones, K., & Waters, E. (2013) · Journal of Public Health · doi:10.1093/pubmed/fdt008 · Cited by 18
Research works printed but not linked: 3
- de Savigny D , Borghi J , Windish R . Chapter 3. Systems thinging: applying a systems perspective to design and evaluate health systems interventions. In: de Savigny D , Tagheed A , eds. Systems thinging for health systems strengtheing. Geneva: World Health Organisation, 2009.
- 10.1177/1356389016652743 · DOI · Paper page
- Montgomery, 2019, Considerations of complexity inrating certainty of evidence insystematic reviews: a primer onusing the grade approach inglobal health, BMJ Glob Health, 4(suppl 1, i78
Cited by
Sorted by citations
Methodological guidance for the conduct of mixed methods systematic reviews
Stern, C., Lizarondo, L. M., Carrier, J., Godfrey, C. M., Rieger, K. L., Salmond, S. W., Apóstolo, J., Kirkpatrick, P., & Loveday, H. P. (2020) · JBI Evidence Synthesis · doi:10.11124/jbisrir-d-19-00169 · Cited by 747
Meeting the review family: exploring review types and associated information retrieval requirements
Sutton, A., Clowes, M., Preston, L., & Booth, A. (2019) · Health Information & Libraries Journal · doi:10.1111/hir.12276 · Cited by 570
Self-citationSkivington, K., Matthews, L., Simpson, S. A., Craig, P., Baird, J., Blazeby, J. M., Boyd, K., Craig, N. J., French, D., McIntosh, E., et al. (2021) · Health Technology Assessment · doi:10.3310/hta25570 · Cited by 585
Flemming, K., Booth, A., Garside, R., Tunçalp, Ö., & Noyes, J. (2019) · BMJ Global Health · doi:10.1136/bmjgh-2018-000882 · Cited by 389
Self-citationQualitative Evidence Synthesis: Where Are We at?
Flemming, K., & Noyes, J. (2021) · International Journal of Qualitative Methods · doi:10.1177/1609406921993276 · Cited by 369
Self-citation
Metadata
- Authors
- Noyes, J., Booth, A., Moore, G., Flemming, K., Tunçalp, Ö., & Shakibazadeh, E.
- Affiliations
- Bangor University (GB)
- University of Sheffield (GB)
- Cardiff University (GB)
- University of York (GB)
- World Health Organization (CH)
- Tehran University of Medical Sciences (IR)
- Corresponding author
- Noyes, Jane
- Venue
- BMJ Global Health
- Publisher
- BMJ
- Published
- 2019-01
- Volume
- 4
- Issue
- Suppl 1
- Pages
- e000893
- ISSN
- 2059-7908 (ISSN-L)
- PMC
- PMC6350750
- OpenAlex
- W2906715785
- Cited by
- 497
- References
- 63
- Citations by year
- 2026: 70 · 2025: 80 · 2024: 83 · 2023: 66 · 2022: 51 · 2021: 83 · 2020: 49 · 2019: 13
- Type
- conference abstract
- Language
- en
- Open access
- gold
- doi.org
- Locations
- BMJ Global Health (publishedVersion, cc-by-nc)
- Europe PMC (PubMed Central) (publishedVersion, cc-by-nc)
- PubMed Central (publishedVersion, cc-by-nc)
- PubMed (publishedVersion)
- White Rose Research Online (University of Leeds, The University of Sheffield, University of York) (acceptedVersion)
- ORCA Online Research @Cardiff (Cardiff University) (acceptedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
- Funders
- Funding provided by the World Health Organization Department of Maternal, Newborn, Child and Adolescent Health through a grant received from the United States Agency for International Development.
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Health Sciences › Health Professions › General Health Professions › Health Policy Implementation Science
- Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Clinical practice guidelines implementation
- Keywords
- complex interventions, evidence synthesis, clinical guideline development, WHO guidelines, complexity perspective
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
