The use of Qualitative Comparative Analysis (QCA) to address causality in complex systems: a systematic review of research on public health interventions

Probably not a substantial claim: may not pass S3 (load-bearing).

Included studies reported as a key strength that QCA provides a method for addressing causal complexity.

“Key strengths reported were that QCA provides a method for addressing causal complexity; and that it provides a systematic approach for understanding the mechanisms at work in implementation across contexts.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Of the full texts, 24 were excluded because they did not report a public health intervention (n=18), had used a methodology inspired by QCA but had not undertaken a QCA (n=2), were protocols or methodological papers only (n=2), or were not published in peer-reviewed journals (n=2).

“Of the full texts, 24 were excluded because: they did not report a public health intervention (n = 18); had used a methodology inspired by QCA, but had not undertaken a QCA (n = 2); were protocols or methodological papers only (n = 2); or were not published in peer-reviewed journals (n = 2) (see Fig. 1).”

Probably not a substantial claim: may not pass S3 (load-bearing).

The majority (n = 19) of included papers were published in public health/health promotion journals.

“The majority (n = 19) were published in public health/health promotion journals”

Probably not a substantial claim: may not pass S3 (load-bearing).

Of the 1384 papers identified after duplicates were removed, 820 were excluded after review of titles and abstracts.

“Of the 1384 papers identified after duplicates were removed, we excluded 820 after review of titles and abstracts (Fig. 1).”

Probably not a substantial claim: may not pass S3 (load-bearing).

The earliest included paper was published in 2005.

“The earliest was published in 2005”

Probably not a substantial claim: may not pass S3 (load-bearing).

Searching the COMPASSS database identified one further study that met the inclusion criteria.

“a suggestion to check the COMPASSS (Comparative Methods for Systematic Cross-Case Analysis) database; this was searched, identifying one further study that met the inclusion criteria [30].”

• cites [30] Harting J, Peters D, Grêaux K, van Assema P, Verweij S, Stronks K, et al. Implementing multiple intervention strategies in Dutch public health-related policy networks. Health Promot Int. 2019;34(2):193–203. https://doi.org/10.1093/heapro/dax067. sourceNot checked yet

  Not checked in the free run.

May combine two claims.

The systematic search identified 1384 papers, of which 27 papers (describing 26 studies) met the inclusion criteria.

“The search identified 1384 papers, of which 27 (describing 26 studies) met the inclusion criteria.”

May combine two claims.

The included studies evaluated interventions across ten public health domains, with the following counts: nutrition/obesity n=8; physical activity n=4; health inequalities n=3; mental health n=2; community engagement n=3; chronic condition management n=3; vaccine adoption or implementation n=2; programme implementation n=3; breastfeeding n=2; general population health n=1.

“Interventions evaluated ranged across: nutrition/obesity (n = 8); physical activity (n = 4); health inequalities (n = 3); mental health (n = 2); community engagement (n = 3); chronic condition management (n = 3); vaccine adoption or implementation (n = 2); programme implementation (n = 3); breastfeeding (n = 2), and general population health (n = 1).”

Probably not a substantial claim: may not pass S3 (load-bearing).

Included studies reported as a key strength that QCA provides a systematic approach for understanding the mechanisms at work in implementation across contexts.

“Key strengths reported were that QCA provides a method for addressing causal complexity; and that it provides a systematic approach for understanding the mechanisms at work in implementation across contexts.”

May combine two claims.

The included studies addressed public health domains including nutrition/obesity (n=8), physical activity (n=4), health inequalities (n=3), mental health (n=2), community engagement (n=3), chronic condition management (n=3), vaccine adoption or implementation (n=2), programme implementation (n=3), breastfeeding (n=2), and general population health (n=1).

“They included nutrition/obesity (n = 8); physical activity (PA) (n = 4); health inequalities (n = 3); mental health (n = 2); community engagement (n = 3); chronic condition management (n = 3); vaccine adoption or implementation (n = 2); programme implementation (n = 3); breastfeeding (n = 2); or general population health (n = 1).”

Probably not a substantial claim: may not pass S3 (load-bearing).

Seventeen of the included papers were published after 2015.

“and 17 were published after 2015.”

May combine two claims.

Discussion of calibration of set membership scores was judged adequate in 15 cases, and inadequate in 11.

“Discussion of calibration of set membership scores was judged adequate in 15 cases, and inadequate in 11”

Probably not a substantial claim: may not pass S3 (load-bearing).

The impossibility of reducing complex conditions to binary conditions was a rationale for using fsQCA in one included study.

“The impossibility of doing this was a rationale for using fsQCA in one study [57]”

May combine two claims.

Several included papers sought to demonstrate the utility of QCA, or to develop QCA methods for health research.

“In addition to the empirical aims, several papers (e.g. [42, 48]) sought to demonstrate the utility of QCA, or to develop QCA methods for health research (e.g. [47]).”

May combine two claims.

Country settings of included studies included any (n=6), OECD countries (n=3), USA (n=6), UK (n=6), and one each from Nepal, Austria, Belgium, Netherlands and Africa.

“Country settings included: any (n = 6); OECD countries (n = 3); USA (n = 6); UK (n = 6) and one each from Nepal, Austria, Belgium, Netherlands and Africa.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Three included studies stated that the QCA was an 'exploratory' analysis.

“Three stated that the QCA was an ‘exploratory’ analysis [53–55].”

May combine two claims.

Assessment of 'familiarity with cases' was made subjectively on the basis of study authors' reports of their knowledge of the settings and the descriptions they provided in the published paper; overall, 14 were judged as sufficient, and 12 less than sufficient.

“Assessment of ‘familiarity with cases’ was made subjectively on the basis of study authors’ reports of their knowledge of the settings (empirical or theoretical) and the descriptions they provided in the published paper: overall, 14 were judged as sufficient, and 12 less than sufficient.”

May combine two claims.

Eleven included studies included additional analyses of the datasets drawn on for the QCA, including six that used qualitative approaches and four reporting additional statistical analyses.

“Eleven studies included additional analyses of the datasets drawn on for the QCA, including six that used qualitative approaches (narrative synthesis, case comparisons), typically to identify cases or conditions for populating the QCA; and four reporting additional statistical analyses (meta-regression, linear regression)”

May combine two claims.

Some included papers noted that QCA is appropriate for addressing causal complexity or multiple pathways to outcome; some noted its appropriateness for providing evidence on how context impacts on interventions; and some noted the need for a method that addressed causal asymmetry.

“including: noting that QCA is appropriate for addressing causal complexity or multiple pathways to outcome [37, 43, 45–51]; noting the appropriateness of the method for providing evidence on how context impacts on interventions [41, 50]; or the need for a method that addressed causal asymmetry [52].”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

First author affiliations of included studies were UK (n=13), USA (n=9), and one each from South Africa, Austria, Belgium, and the Netherlands.

“These largely reflected the first author’s country affiliations in the UK (n = 13); USA (n = 9); and one each from South Africa, Austria, Belgium, and the Netherlands.”

May combine two claims.

Weaknesses reported in included studies included limitations of QCA in general for addressing complexity, as well as specific limitations with either the csQCA or the fsQCA methods employed.

“Weaknesses reported included the limitations of QCA in general for addressing complexity, as well as specific limitations with either the csQCA or the fsQCA methods employed.”

May combine two claims.

Only five included studies met all six quality criteria (evidence of familiarity with cases, justification of case selection, discussion of calibration, reporting truth tables, reporting raw data matrices, reporting coverage and consistency); a further six met at least five of them.

“Only five studies met all six of these quality criteria (evidence of familiarity with cases, justification of case selection, discussion of calibration, reporting truth tables, reporting raw data matrices, reporting coverage and consistency); a further six met at least five of them.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Potential limitations (especially in csQCA studies) of reducing complex conditions, drawn from detailed qualitative understanding, to binary conditions were noted in included studies.

“as were the potential limitations (especially in csQCA studies) of reducing complex conditions, drawn from detailed qualitative understanding, to binary conditions [35].”

May combine two claims.

Sample sizes of included studies ranged from 10 to 131, with no study having small N (<10) sample sizes, four having large N (>50) sample sizes, and the majority (22) being medium N studies (in the range 10-50).

“Sample sizes ranged from 10 to 131, with no study having small N (< 10) sample sizes, four having large N (> 50) sample sizes, and the majority (22) being medium N studies (in the range 10–50).”

Probably not a substantial claim: may not pass S3 (load-bearing).

Included studies noted that QCA does not deal with probabilistic causation.

“That QCA does not deal with probabilistic causation was noted [47].”

May combine two claims.

Included studies highlighted strengths of QCA as providing possibilities for examining complex policy problems; for testing existing as well as new theory; and for identifying aspects of interventions which had not been previously perceived as critical or which may have been missed when drawing on statistical methods that use linear additive models.

“Authors highlighted the strengths of QCA as providing possibilities for examining complex policy problems [37, 59]; for testing existing as well as new theory [52]; and for identifying aspects of interventions which had not been previously perceived as critical [41] or which may have been missed when drawing on statistical methods that use, for instance, linear additive models [42].”

May combine two claims.

Included studies reported that QCA was a useful tool to address complex causality, providing a systematic approach to understand the mechanisms at work in implementation across contexts.

“Specifically, that it was a useful tool to address complex causality, providing a systematic approach to understand the mechanisms at work in implementation across contexts”

Probably not a substantial claim: may not pass S3 (load-bearing).

Some included studies made the case that csQCA provides more appropriate findings for policy because dichotomisation forces a focus on meaningful distinctions, including those related to decisions that practitioners/policy makers can action.

“However, others [47] make the case that csQCA provides more appropriate findings for policy: dichotomisation forces a focus on meaningful distinctions, including those related to decisions that practitioners/policy makers can action.”

May combine two claims.

Twelve included studies drew on primary data collected specifically for the study, with five of those additionally drawing on secondary data sets; five drew only on secondary data sets, and nine used data from systematic reviews of published research.

“Twelve studies drew on primary data collected specifically for the study, with five of those additionally drawing on secondary data sets; five drew only on secondary data sets, and nine used data from systematic reviews of published research.”

May combine two claims.

The largest number of included studies (n=10), including all the systematic reviews, analysed cases at the level of the intervention or a component of the intervention; seven analysed organisational level cases; five analysed sub-national region level cases; and two each analysed country or individual level cases.

“The largest number of studies (n = 10), including all the systematic reviews, analysed cases at the level of the intervention, or a component of the intervention; seven analysed organisational level cases (e.g. school class, network, primary care practice); five analysed sub-national region level cases (e.g. state, local authority area), and two each analysed country or individual level cases.”

May combine two claims.

Fifteen included studies used crisp sets (csQCA), and 11 used fuzzy sets (fsQCA).

“Fifteen studies used crisp sets (csQCA), 11 used fuzzy sets (fsQCA).”

Probably not a substantial claim: may not pass S3 (load-bearing).

One included study used an additional Boolean configurational technique to reduce the number of conditions in the QCA analysis.

“One study used an additional Boolean configurational technique to reduce the number of conditions in the QCA analysis [40].”

May combine two claims.

High-income countries dominate in both intervention settings and author affiliations of included studies.

“The dominance of high-income countries in both intervention settings and author affiliations is disappointing”

May combine two claims.

The two larger N (>100 cases) studies in the sample were an individual level analysis of survey data and an analysis of intervention arms from a systematic review.

“The two larger N (> 100 cases) studies in our sample were an individual level analysis of survey data [46, 47] and an analysis of intervention arms from a systematic review [50].”

May combine two claims.

This review demonstrated the large range of evaluation questions that have been addressed to date using QCA, including contributions to process evaluations of trials and for exploring conditions leading to effectiveness or not in systematic reviews of interventions.

“This review has demonstrated the large range of evaluation questions that have been addressed to date using QCA, including contributions to process evaluations of trials and for exploring the conditions leading to effectiveness (or not) in systematic reviews of interventions.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Uncertain inclusions were checked by the third author (MP).

“Uncertain inclusions were checked by the third author (MP).”

Probably not a substantial claim: may not pass S3 (load-bearing).

Double screening of all records was assisted by EPPI Reviewer 4's machine learning function.

“Double screening of all records was assisted by EPPI Reviewer 4’s machine learning function.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Two papers relating to the same research question and dataset were combined, such that analysis was by study (n=26) not by paper.

“Two papers relating to the same research question and dataset were combined, such that analysis was by study (n = 26) not by paper.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

A protocol was prepared in accordance with the PRISMA-P 2015 statement and published in 2019.

“A protocol was prepared in accordance with the Preferred Reporting Items for Systematic Reviews and Meta-Analysis Protocols (PRISMA-P) 2015 statement [28], and published in 2019 [27], where the methods are explained in detail.”

• cites [28] Shamseer L, Moher D, Clarke M, Ghersi D, Liberati A, Petticrew M, et al. Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015: elaboration and explanation. BMJ. 2015;349(1):g7647. https://doi.org/10.1136/bmj.g7647. sourceNot checked yet

  Not checked in the free run.

• cites [27] Hanckel B, Petticrew M, Thomas J, Green J. Protocol for a systematic review of the use of qualitative comparative analysis for evaluative questions in public health research. Syst Rev. 2019;8(1):252. https://doi.org/10.1186/s13643-019-1159-5. sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

The list of included studies was sent to experts in QCA methods in health and related fields, including authors of included studies and/or those who had published on QCA methodology.

“The list of included studies was sent to experts in QCA methods in health and related fields, including authors of included studies and/or those who had published on QCA methodology.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Full texts of the 51 remaining studies were screened by BH and JG for inclusion, with 10 papers double coded by both authors, with complete agreement.

“Full texts of the 51 remaining studies were screened by BH and JG for inclusion, with 10 papers double coded by both authors, with complete agreement.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The systematic review was registered with PROSPERO on 29 April 2019 (CRD42019131910).

“This systematic review was registered with the International Prospective Register of Systematic Reviews (PROSPERO) on 29 April 2019 (CRD42019131910).”

Probably not a substantial claim: may not pass S3 (load-bearing).

There is a potential trade-off in providing 'interpretable results', but ones which preclude potential for utilising more detailed information.

“There is, then, a potential trade-off in providing ‘interpretable results’, but ones which preclude potential for utilising more detailed information [45].”

Classified as probably not a substantial claim.

QCA has been widely used in public health research over the last decade to advance understanding of causal inference in complex systems.

“QCA has been widely used in public health research over the last decade to advance understanding of causal inference in complex systems.”

Probably not a substantial claim: may not pass S3 (load-bearing).

There is increasing interest in QCA as an alternative or complementary technique to regression-oriented statistical methods for larger samples.

“However, there is increasing interest in the method as an alternative or complementary technique to regression-oriented statistical methods for larger samples [65]”

• cites [65] Fiss PC, Sharapov D, Cronqvist L. Opposites attract? Opportunities and challenges for integrating large-N QCA and econometric analysis. Polit Res Q. 2013;66:191–8.Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

A limitation of the review is the inevitable difficulty in operationalising a 'public health' intervention.

“However, a limitation is the inevitable difficulty in operationalising a ‘public health’ intervention.”

May combine two claims.

As the number of conditions included increases, so too does the number of possible configurations, increasing the chance of unique combinations and of generating spurious solutions with a high level of consistency.

“As the number of conditions included increases, so too does the number of possible configurations, increasing the chance of unique combinations and of generating spurious solutions with a high level of consistency.”

Probably not a substantial claim: may not pass S3 (load-bearing).

There are growing number of descriptive epidemiological studies using QCA to explore factors predicting outcomes across diverse areas.

“Similarly, there are growing number of descriptive epidemiological studies using QCA to explore factors predicting outcomes across such diverse areas as lupus and quality of life [68]; length of hospital stay [69]; constellations of factors predicting injury [70]; or the role of austerity, crisis and recession in predicting public health outcomes [71].”

• cites [68] Rioja EC. Valero-Moreno S, Giménez-Espert M del C, Prado-Gascó V. the relations of quality of life in patients with lupus erythematosus: regression models versus qualitative comparative analysis. J Adv Nurs. 2019;75(7):1484–92. https://doi.org/10.1111/jan.13957. sourceNot checked yet

  Not checked in the free run.

• cites [69] Dy SM. Garg Pushkal, Nyberg Dorothy, Dawson Patricia B., Pronovost Peter J., Morlock Laura, et al. critical pathway effectiveness: assessing the impact of patient, hospital care, and pathway characteristics using qualitative comparative analysis. Health Serv Res. 2005;40(2):499–516. https://doi.org/10.1111/j.1475-6773.2005.0r370.x. sourceNot checked yet

  Not checked in the free run.

• cites [70] MELINDER KA, ANDERSSON R. The impact of structural factors on the injury rate in different European countries. Eur J Pub Health. 2001;11(3):301–8. https://doi.org/10.1093/eurpub/11.3.301. sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S2 (checkable).

Exclusions on scope are not straightforward, given that most social, environmental and political conditions impact on public health.

“Exclusions on scope are not straightforward, given that most social, environmental and political conditions impact on public health”

• AssumptionRests on an assumption: Most social, environmental and political conditions impact on public health.

Probably not a substantial claim: may not pass S3 (load-bearing).

Methodological debate on QCA is still thriving on issues such as the reliability of causal inferences and the usefulness of the method for policy decisions.

“methodological debate is still thriving on issues such as the reliability of causal inferences [72], alongside more general critiques of the usefulness of the method for policy decisions (see, for instance, [73]).”

• cites [72] Baumgartner M, Thiem A. Often trusted but never (properly) tested: evaluating qualitative comparative analysis. Sociol Methods Res. 2020;49(2):279–311. https://doi.org/10.1177/0049124117701487. sourceNot checked yet

  Not checked in the free run.

• cites [73] Tanner S. QCA is of questionable value for policy research. Polic Soc. 2014;33(3):287–98. https://doi.org/10.1016/j.polsoc.2014.08.003. sourceNot checked yet

  Not checked in the free run.

May combine two claims.

Research teams conducting QCA in the context of systematic reviews may have detailed understanding in general of interventions within their spheres of expertise, but they are unlikely to have this for the whole range of cases.

“Whilst research teams conducting QCA in the context of systematic reviews may have detailed understanding in general of interventions within their spheres of expertise, they are unlikely to have this for the whole range of cases”

Probably not a substantial claim: may not pass S3 (load-bearing).

Researchers drawing on evidence from systematic reviews for their data are reliant largely on published evidence for information on conditions.

“First, researchers drawing on evidence from systematic reviews for their data are reliant largely on published evidence for information on conditions”

Probably not a substantial claim: may not pass S3 (load-bearing).

The dominance of high-income countries in included studies reflects the disproportionate location of public health research in the global north more generally.

“but reflects the disproportionate location of public health research in the global north more generally [63].”

• cites [63] Plancikova D, Duric P, O’May F. High-income countries remain overrepresented in highly ranked public health journals: a descriptive analysis of research settings and authorship affiliations. Crit Public Health 2020;0:1–7, DOI: https://doi.org/10.1080/09581596.2020.1722313. sourceNot checked yet

  Not checked in the free run.

Classified as probably not a substantial claim.

QCA is primarily seen as a method that requires thick, detailed knowledge of cases, particularly when the aim is to understand complex causation.

“given that QCA is primarily seen as a method that requires thick, detailed knowledge of cases, particularly when the aim is to understand complex causation [8].”

• cites [8] Gerrits L, Pagliarin S. Social and causal complexity in qualitative comparative analysis (QCA): strategies to account for emergence. Int J Soc Res Methodol 2020;0:1–14, doi: https://doi.org/10.1080/13645579.2020.1799636. sourceNot checked yet

  Not checked in the free run.

Classified as probably not a substantial claim. May combine two claims.

The review did not assess quality directly but used indicators of good practice from QCA methodological literature, largely written for policy studies scholars and often post-dating the included studies.

“We did not assess quality directly, but used indicators of good practice discussed in QCA methodological literature, largely written for policy studies scholars, and often post-dating the publication dates of studies included in this review.”

May combine two claims.

Restricting the search to English and peer-reviewed publications may have missed bodies of work from many regions and excluded research from non-governmental organisations using QCA methods in evaluation.

“Restricting our search to publications in English and to peer reviewed publications may have missed bodies of work from many regions, and has excluded research from non-governmental organisations using QCA methods in evaluation.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The choice of evaluation design depends in part on availability of counterfactuals, investigator control over the intervention, range of potential cases and contexts, and political considerations such as credibility with stakeholders.

“As in other fields, the choice of design will in part depend on the availability of counterfactuals, the extent to which the investigator can control the intervention, and the range of potential cases and contexts [75], as well as political considerations, such as the credibility of the approach with key stakeholders [76].”

• cites [75] Stern E, Stame N, Mayne J, Forss K, Davies R, Befani B. Broadening the range of designs and methods for impact evaluations. Technical report. London: DfiD; 2012. sourceNot checked yet

  Not checked in the free run.

• cites [76] Pattyn V. Towards appropriate impact evaluation methods. Eur J Dev Res. 2019;31(2):174–9. https://doi.org/10.1057/s41287-019-00202-w. sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.

QCA will not be appropriate for all evaluations and cannot be the only answer to addressing complex causality.

“However, QCA will not be appropriate for all evaluations, and cannot be the only answer to addressing complex causality.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Craig et al. noted that qualitative techniques, the cited references for which are all QCA studies, require further studies to establish their validity and usefulness.

“noting only that what they call ‘qualitative’ techniques (the cited references for these are all QCA studies) require “further studies … to establish their validity and usefulness” [2].”

• cites [2] Craig P, Katikireddi SV, Leyland A, Popham F. Natural experiments: an overview of methods, approaches, and contributions to public health intervention research. Annu Rev Public Health. 2017;38(1):39–56. https://doi.org/10.1146/annurev-publhealth-031816-044327. sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.

There is potential for QCA to be more widely used in evaluative research, to identify conditions under which interventions across contexts are implemented or not, and configurations of conditions associated with effect or lack of evidence of effect.

“There is potential for QCA to be more widely used in evaluative research, to identify the conditions under which interventions across contexts are implemented or not, and the configurations of conditions associated with effect or lack of evidence of effect.”

Classified as probably not a substantial claim.

This review was limited to public health evaluation research.

“This review was limited to public health evaluation research: other domains that would be useful to map include health systems/services interventions and studies used to design or target interventions.”

Classified as probably not a substantial claim. May combine two claims.

QCA is a relatively new approach for public health research, with promise for contributing to methodological development for addressing causation in complex systems.

“QCA is a relatively new approach for public health research, with promise for contributing to much-needed methodological development for addressing causation in complex systems.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

Given limitations in the evidence base on what works to address inequalities in health, QCA has potential to help identify conditions under which interventions do or do not exacerbate unequal outcomes, or conditions leading to differential uptake or impacts across sub-population groups.

“Given the limitations in the evidence base on what works to address inequalities in health, for instance [74], QCA has potential here, to help identify the conditions under which interventions do or do not exacerbate unequal outcomes, or the conditions that lead to differential uptake or impacts across sub-population groups.”

• cites [74] Mackenbach JP. Tackling inequalities in health: the need for building a systematic evidence base. J Epidemiol Community Health. 2003;57(3):162. https://doi.org/10.1136/jech.57.3.162. sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

As a stand-alone approach, QCA might be particularly valuable for national and regional comparative studies of the impact of policies on public health outcomes.

“As a stand-alone approach, QCA might be particularly valuable for national and regional comparative studies of the impact of policies on public health outcomes.”

May combine two claims.

QCA is a feasible or appropriate method for public health evaluation when there are sufficient comparable cases with and without the outcome of interest and researchers have sufficiently in-depth understanding of those cases to make credible calibration decisions.

“The studies included in this review have demonstrated that QCA is a feasible method when there are sufficient (comparable) cases for identifying configurations of conditions under which interventions are effective (or not), or are implemented (or not).”

• Also stated in: § Implications for future research · hedged

  “The evidence from this review suggests that QCA evaluation approaches are feasible when there is a sufficient number of comparable cases with and without the outcome of interest, and when the investigators have, or can generate, sufficiently in-depth understanding of those cases to make sense of connections between conditions, and to make credible decisions about the calibration of set membership.”

• Also stated in: § Conclusions · plain

  “For explanatory questions, the approach is most appropriate when there is a series of enough comparable cases with and without the outcome of interest, and where the researchers have detailed understanding of those cases, and conditions.”

Probably not a substantial claim: may not pass S4 (based).

Alongside cluster randomised trials or systematic reviews, QCA approaches are especially useful for identifying core combinations of causal conditions for success and lack of success in implementation and outcome.

“Alongside cluster randomised trials of interventions, or alongside systematic reviews, QCA approaches are especially useful for identifying core combinations of causal conditions for success and lack of success in implementation and outcome.”

Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.

QCA may be particularly relevant for understanding multiple causation, where different configurations might lead to the same outcome, and for understanding conditions associated with both lack of effect and effect.

“QCA may be particularly relevant for understanding multiple causation (that is, where different configurations might lead to the same outcome), and for understanding the conditions associated with both lack of effect and effect.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

Validating the findings of a QCA study is perhaps as challenging as validating findings from any other design, given the absence of gold standard comparators.

“Validating the findings of a QCA study is perhaps as challenging as validating the findings from any other design, given the absence of any gold standard comparators.”

• AssumptionRests on an assumption: There is no gold standard comparator for validating findings from QCA or other designs.