Guarding Against False Positives in Qualitative Comparative Analysis
Open accessThe DOI matches its Crossref record; 21 of the 24 references it cites were found in a registry or an index.
Abstract
The various methodological techniques that fall under the umbrella description of qualitative comparative analysis (QCA) are increasingly popular for modeling causal complexity and necessary or sufficient conditions in medium-N settings. Because QCA methods are not designed as statistical techniques, however, there is no way to assess the probability that the patterns they uncover are the result of chance. Moreover, the implications of the multiple hypothesis tests inherent in these techniques for the false positive rate of the results are not widely understood. This article fills both gaps by tailoring a simple permutation test to the needs of QCA users and adjusting the Type I error rate of the test to take into account the multiple hypothesis tests inherent in QCA. An empirical application–a reexamination of a study of protest-movement success in the Arab Spring–highlights the need for such a test by showing that even very strong QCA results may plausibly be the result of chance.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Nevertheless, this is also a drawback of QCA since it can give false positives, or in other words, be the result of chance (for details, see Braumoeller [2015]).”
From A qualitative assessment of QCA: method stretching in large-N studies and temporality (2022) · doi:10.1007/s11135-021-01278-5 · cites this paper as Braumoeller 2015
Citations
Cites
Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple Testing
Benjamini, Y., & Hochberg, Y. (1995) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1995.tb02031.x · Cited by 109,482 · cited 1×
“The most straightforward way to do so 10Jirka Lewandowski (private communication) has argued in favor of using the false discovery rate (FDR; see Benjamini and Hochberg 1995) rather than the FWER.”
BackgroundA Simple Sequentially Rejective Multiple Test Procedure
(1979) · Scandinavian Journal of Statistics · doi:10.2307/4615733 · Cited by 21,526
Redesigning Social Inquiry: Fuzzy Sets and Beyond
Ragin, C. C. (2008) · University of Chicago Press eBooks · doi:10.7208/chicago/9780226702797.001.0001 · Cited by 4,445
Set-Theoretic Methods for the Social Sciences: A Guide to Qualitative Comparative Analysis
Schneider, C. Q., & Wagemann, C. (2012) · doi:10.1017/cbo9781139004244 · Cited by 2,862
Why Permutation Tests Are Superior to t and F Tests in Biomedical Research
Ludbrook, J., & Dudley, H. (1998) · The American Statistician · doi:10.2307/2685470 · Cited by 281
Marx, A., & Dusa, A. (2011) · Methodological Innovations Online · doi:10.4256/mio.2010.0037 · Cited by 273
Evidence and Inference in the Comparative Case Study
Dion, D. (1998) · Comparative Politics · doi:10.2307/422284 · Cited by 261
The Methodology of Necessary Conditions
Braumoeller, B. F., & Goertz, G. (2000) · American Journal of Political Science · doi:10.2307/2669285 · Cited by 255
Self-citationWhat Best Explains Successful Protest Cascades? ICTs and the Fuzzy Causes of the Arab Spring
Hussain, M. M., & Howard, P. N. (2013) · International Studies Review · doi:10.1111/misr.12020 · Cited by 191
Hug, S. (2013) · Political Analysis · doi:10.1093/pan/mps061 · Cited by 137 · cited 3×
“Moreover, as Hug (2013) points out, csQCA’s sensitivity to counterexamples makes it vulnerable to even a small amount of measurement error.”
BackgroundInfluentialFuzzy Sets on Shaky Ground: Parameter Sensitivity and Confirmation Bias in fsQCA
Krogslund, C., Choi, D. D., & Poertner, M. (2015) · Political Analysis · doi:10.1093/pan/mpu016 · Cited by 136 · cited 1×
“For example, Krogslund, Choi, and Poertner (2015) reexamine three prominent studies that use fsQCA and find that completely random variables added to the model specifications are identified as part of at least one sufficient condition quite often—from 75.2% of the time to 100% of the time,…”
BackgroundInfluentialCronqvist, L., & Berg‐Schlosser, D. (2009) · Sage eBooks · doi:10.4135/9781452226569.n4 · Cited by 136
The Youth Bulge in Egypt: An Intersection of Demographics, Security, and the Arab Spring
LaGraffe, D. (2012) · Journal of Strategic Security · doi:10.5038/1944-0472.5.2.4 · Cited by 135
Doing Justice to Logical Remainders in QCA: Moving beyond the Standard Analysis
Schneider, C. Q., & Wagemann, C. (2013) · Political Research Quarterly · Cited by 82
ABOUT A REVOLUTION: THE ECONOMIC MOTIVATIONS OF THE ARAB SPRING
ANSANI, A., & Daniele, V. (2012) · International Journal of Development and Conflict · doi:10.1142/s2010269012500135 · Cited by 74
Introductory Statistics for the Behavioral Sciences
(1971) · Elsevier eBooks · doi:10.1016/c2013-0-11685-6 · Cited by 68
Goodness-of-Fit Tests and Descriptive Measures in Fuzzy-Set Analysis
Eliason, S. R., & Stryker, R. (2009) · Sociological Methods & Research · doi:10.1177/0049124109339371 · Cited by 61
Ceilings and Floors: Where Are There No Observations?
Goertz, G., Hak, T., & Dul, J. (2012) · Sociological Methods & Research · doi:10.1177/0049124112460375 · Cited by 47
Beaumont, G. P. (2005) · Elsevier eBooks · doi:10.1533/9780857099471.268 · Cited by 4
A Broad Approach to Countering the Islamic State
Paul, C., & Clarke, C. P. (2014) · Cited by 2
Replication Data for: "Guarding Against False Positives in Qualitative Comparative Analysis"
Braumoeller, B. F. (2015) · Harvard Dataverse · doi:10.7910/dvn/gy6p9i · Cited by 1
Self-citation
Research works printed but not linked: 8
- Good, 2005, Permutation, parametric and bootstrap tests of hypotheses, Springer Series in Statistics.
- Ragin, 2004, Between complexity and parsimony: Limited diversity, counterfactual cases, and comparative analysis.
- Rihoux, 2013, From niche to mainstream method? A comprehensive mapping of QCA applications in journal articles from 1984 to 2011, Political Research Quarterly, 66
- Ragin, 2008, User's guide to fuzzy-set/qualitative comparative analysis
- Doyle, 1983a, Kant, liberal legacies, and foreign affairs, Philosophy and Public Affairs, 12
- Ragin, 2000, Fuzzy-set social science.
- Doyle, 1983b, Kant, liberal legacies, and foreign affairs, part 2, Philosophy and Public Affairs, 12
- Ragin, 1987, The comparative method: Moving beyond qualitative and quantitative strategies.
Cited by
A Statistical Significance Test for Necessary Condition Analysis
Dul, J., van der Laan, E., & Kuik, R. (2020) · Organizational Research Methods · doi:10.1177/1094428118795272 · Cited by 675
“…fast computers, permutation tests have been developed for correlation and regression (Anderson & Robinson, 2001; DiCiccio & Romano, 2017), ANOVA (Anderson, 2001), the general linear model (Winkler, Ridgway, Webster, Smith, & Nichols, 2014), and qualitative comparative analysis (Braumoeller, 2015).”
MethodThe contributions of qualitative comparative analysis (QCA) to international business research
Fainshmidt, S., Witt, M. A., Aguilera, R. V., & Verbeke, A. (2020) · Journal of International Business Studies · doi:10.1057/s41267-020-00313-1 · Cited by 452
“There are indicators of the quality of a QCA model, as discussed earlier, but the scholars who originally developed QCA did not have in mind the concept of ‘giving a probability that a particular relationship does not arise by chance’ (see Braumoeller, 2015 for further analysis).”
An overview of qualitative comparative analysis: A bibliometric analysis
Roig-Tierno, N., Gonzalez-Cruz, T. F., & Llopis-Martinez, J. (2017) · Journal of Innovation & Knowledge · doi:10.1016/j.jik.2016.12.002 · Cited by 392
“According to Braumoeller (2015), the solution to the false positives is an adaptation to a swap test of QCA’s requirements, by adjusting the error Type 1 index (false positive) of each individual test in order to include the multiple hypotheses tests and assuring a FWER (family-wise error rate) in…”
BackgroundMethodDesigning Research With Qualitative Comparative Analysis (QCA): Approaches, Challenges, and Tools
(2017) · doi:10.1177/0049124117729700 · Cited by 380
Leaders and Laggards: Climate Policy Ambition in Developed States
Tobin, P. (2017) · Global Environmental Politics · doi:10.1162/glep_a_00433 · Cited by 173
Metadata
- Authors
- Braumoeller, B. F.
- Affiliations
- The Ohio State University (US)
- Corresponding author
- Braumoeller, Bear F.
- ORCID
- Braumoeller, Bear F.
- Venue
- Political Analysis
- Publisher
- Cambridge University Press (CUP)
- Published
- 2015
- Volume
- 23
- Issue
- 4
- Pages
- 471-487
- ISSN
- 1047-1987 (ISSN-L)
- 1476-4989
- OpenAlex
- W2264462533
- Semantic Scholar
- c333e7ea53bf3e7e5afb6b1dd9d35e95d6f396d7
- Cited by
- 77
- References
- 24
- Citations by year
- 2026: 7 · 2025: 10 · 2024: 4 · 2023: 9 · 2022: 9 · 2021: 8 · 2020: 5 · 2019: 6 · 2018: 7 · 2017: 5 · 2016: 3 · 2015: 4
- FWCI
- 20.96
- Type
- journal article
- Language
- en
- Open access
- green
- papers.ssrn.com
- Locations
- Political Analysis (publishedVersion)
- SSRN Electronic Journal (submittedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- Topics
- Social Sciences › Sociology and Political Science › Qualitative Comparative Analysis Research
- Social Sciences › Political Science and International Relations › Political Science Research and Education
- Social Sciences › General Social Sciences › Computational and Text Analysis Methods
- Keywords
- qualitative comparative analysis, false positives, permutation tests, type I error rate, multiple comparisons, necessary and sufficient conditions, causal complexity, Arab Spring
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.