Often Trusted but Never (Properly) Tested: Evaluating Qualitative Comparative Analysis
Abstract
To date, hundreds of researchers have employed the method of Qualitative Comparative Analysis (QCA) for the purpose of causal inference. In a recent series of simulation studies, however, several authors have questioned the correctness of QCA in this connection. Some prominent representatives of the method have replied in turn that simulations with artificial data are unsuited for assessing QCA. We take issue with either position in this impasse. On the one hand, we argue that data-driven evaluations of the correctness of a procedure of causal inference require artificial data. On the other hand, we prove all previous attempts in this direction to have been defective. For the first time in the literature on configurational comparative methods, we lay out a set of formal criteria for an adequate evaluation of QCA before implementing a battery of inverse-search trials to test how this method performs in different recovery contexts according to these criteria. While our results indicate that QCA is correct when generating the parsimonious solution type, they also demonstrate that the method is incorrect when generating the conservative and intermediate solution type. In consequence, researchers using QCA for causal inference, particularly in human-sensitive areas such as public health and medicine, should immediately discontinue employing the method’s conservative and intermediate search strategies.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Methodological debate on QCA is still thriving on issues such as the reliability of causal inferences and the usefulness of the method for policy decisions.
“methodological debate is still thriving on issues such as the reliability of causal inferences [72], alongside more general critiques of the usefulness of the method for policy decisions (see, for instance, [73]).”
From The use of Qualitative Comparative Analysis (QCA) to address causality in complex systems: a systematic review of research on public health interventions (2021) · cites this paper as [72]
Cites
Data Analysis Using Regression and Multilevel/Hierarchical Models
Gelman, A., & Hill, J. (2006) · doi:10.1017/cbo9780511790942 · Cited by 13,597 · see citing works
Causality: Models, Reasoning, and Inference
Pearl, J. (2009) · doi:10.1017/cbo9780511803161 · Cited by 11,653 · see citing works
Grossman, P. (2009) · doi:10.1007/978-0-230-37405-8_1 · Cited by 8,906 · see citing works
Journal of The American Statistical Association
Fienberg, S. E. (2004) · Encyclopedia of Statistical Sciences · doi:10.1002/0471667196.ess1317 · Cited by 6,289 · see citing works
Statistics and Causal Inference
Holland, P. W. (1986) · Journal of the American Statistical Association · doi:10.1080/01621459.1986.10478354 · Cited by 5,293 · see citing works
Ragin, C. C. (2008) · University of Chicago Press eBooks · doi:10.7208/chicago/9780226702797.001.0001 · Cited by 4,595 · see citing works
Causation, Prediction, and Search
Spirtes, P., Glymour, C., & Scheines, R. (2001) · doi:10.7551/mitpress/1754.001.0001 · Cited by 4,521 · see citing works
An axiomatic basis for computer programming
Hoare, C. A. R. (1969) · Communications of the ACM · doi:10.1145/363235.363259 · Cited by 4,006 · see citing works
Set-Theoretic Methods for the Social Sciences: A Guide to Qualitative Comparative Analysis
Schneider, C. Q., & Wagemann, C. (2012) · doi:10.1017/cbo9781139004244 · Cited by 2,862 · see citing works
Causal inference in statistics: An overview
Pearl, J. (2009) · Statistics Surveys · doi:10.1214/09-ss057 · Cited by 2,336 · cited 4× · see citing works
“We argue not only in agreement with Lucas and Sza-trowski (2014), but, what is more, also the vast literature on causal inference, that rigorous testing requires simulated data (e.g., Hyttinen, Eberhardt, and Hoyer 2012; Kleinberg 2013; Morgan and Winship 2007; Pearl 2009b).”
BackgroundMethodResultInfluentialAdvances in Neural Information Processing Systems 21
Griffiths, T. L., Lucas, C. G., Williams, J. J., & Kalish, M. L. (2009) · neural information processing systems · Cited by 1,854 · see citing works
Nature's capacities and their measurement
(1990) · Choice Reviews Online · doi:10.5860/choice.27-4463 · Cited by 1,347 · see citing works
Timberlake, M., & Ragin, C. C. (1989) · Social Forces · doi:10.2307/2579563 · Cited by 1,313 · see citing works
Counterfactuals and Causal Inference: Methods and Principles for Social Research
Morgan, S. L., & Winship, C. (2007) · doi:10.1017/cbo9780511804564 · Cited by 1,146 · see citing works
Diamond, A., & Sekhon, J. S. (2012) · The Review of Economics and Statistics · doi:10.1162/rest_a_00318 · Cited by 1,066 · see citing works
The cement of the universe, a study of causation
Rota, G. (1978) · Advances in Mathematics · doi:10.1016/0001-8708(78)90103-2 · Cited by 698 · see citing works
Rihoux, B. (2006) · International Sociology · doi:10.1177/0268580906067836 · Cited by 461 · see citing works
Cress, D. M., & Snow, D. A. (1996) · American Sociological Review · doi:10.2307/2096310 · Cited by 457 · see citing works
Combining QCA and Process Tracing in Set-Theoretic Multi-Method Research
Schneider, C. Q., & Rohlfing, I. (2013) · Sociological Methods & Research · doi:10.1177/0049124113481341 · Cited by 385 · see citing works
Schneider, C. Q., & Wagemann, C. (2010) · Comparative Sociology · doi:10.1163/156913210x12493538729838 · Cited by 308 · see citing works
Boolos, G., Burgess, J. P., & Jeffrey, R. C. (2007) · Cambridge University Press eBooks · doi:10.1017/cbo9780511804076 · Cited by 301 · see citing works
The Case for Qualitative Comparative Analysis (QCA): Adding Leverage for Thick Cross-Case Comparison
Rihoux, B., & Lobe, B. (2009) · SAGE Publications Ltd eBooks · doi:10.4135/9781446249413.n13 · Cited by 257 · see citing works
QCA: A Package for Qualitative Comparative Analysis
Thiem, A., & Duşa, A. (2013) · The R Journal · doi:10.32614/rj-2013-009 · Cited by 197 · see citing works
Self-citationMendel, J. M., & Korjani, M. M. (2012) · Information Sciences · doi:10.1016/j.ins.2012.02.039 · Cited by 156 · see citing works
Qualitative Comparative Analysis in Critical Perspective
Lucas, S. R., & Szatrowski, A. (2014) · Sociological Methodology · doi:10.1177/0081175014532763 · Cited by 150 · see citing works
Model Ambiguities in Configurational Comparative Research
Baumgärtner, M., & Thiem, A. (2015) · Sociological Methods & Research · doi:10.1177/0049124115610351 · Cited by 149 · see citing works
Self-citationBaumgärtner, M. (2014) · Quality & Quantity · doi:10.1007/s11135-014-0026-7 · Cited by 144 · see citing works
Qualitative Comparative Analysis with R
Thiem, A., & Duşa, A. (2012) · SpringerBriefs in political science · doi:10.1007/978-1-4614-4584-5 · Cited by 144 · see citing works
Self-citationThiem, A., Baumgartner, M., & Bol, D. (2016) · Comparative Political Studies · doi:10.1177/0010414014565892 · Cited by 138 · see citing works
Self-citationHug, S. (2013) · Political Analysis · doi:10.1093/pan/mps061 · Cited by 137 · cited 2× · see citing works
“14 Lastly, critics have maintained that QCA solutions tend not to be robust (Hug 2013; Krogslund et al. 2015; Lucas and Szatrowski 2014).”
BackgroundFuzzy Sets on Shaky Ground: Parameter Sensitivity and Confirmation Bias in fsQCA
Krogslund, C., Choi, D. D., & Poertner, M. (2015) · Political Analysis · doi:10.1093/pan/mpu016 · Cited by 136 · cited 3× · see citing works
“14 Lastly, critics have maintained that QCA solutions tend not to be robust (Hug 2013; Krogslund et al. 2015; Lucas and Szatrowski 2014).”
BackgroundQualitative comparative analysis vis-à-vis regression
Seawright, J. (2005) · Studies in Comparative International Development · doi:10.1007/bf02686284 · Cited by 134 · see citing works
The Programmatic Emergence of the Social Security State
Hicks, A., Misra, J., & Ng, T. N. (1995) · American Sociological Review · doi:10.2307/2096417 · Cited by 131 · see citing works
A Simple Multivariate Test for Asymmetric Hypotheses
Clark, W. R., Gilligan, M. J., & Golder, M. (2006) · Political Analysis · doi:10.1093/pan/mpj018 · Cited by 120 · see citing works
Cronqvist, L., & Berg‐Schlosser, D. (2009) · Sage eBooks · doi:10.4135/9781452226569.n4 · Cited by 118 · see citing works
Baumgärtner, M. (2009) · Sociological Methods & Research · doi:10.1177/0049124109339369 · Cited by 101 · see citing works
Qualitative & Multi-Method Research
Tanner, S. (2014) · Cited by 101 · see citing works
A Qualitative Comparative Analysis of Conversion to Venezuelan Evangelicalism: How Networks Matter
Smilde, D. A. (2005) · American Journal of Sociology · doi:10.1086/497306 · Cited by 97 · see citing works
Thiem, A., & Duşa, A. (2013) · Social Science Computer Review · doi:10.1177/0894439313478999 · Cited by 95 · see citing works
Self-citationLearning linear cyclic causal models with latent variables
Hyttinen, A., Eberhardt, F., & Hoyer, P. O. (2012) · Journal of Machine Learning Research · doi:10.5555/2503308.2503350 · Cited by 90 · see citing works
Causality, Probability, and Time
Kleinberg, S. (2012) · Cambridge University Press eBooks · doi:10.1017/cbo9781139207799 · Cited by 82 · see citing works
Conducting Configurational Comparative Research With Qualitative Comparative Analysis
Thiem, A. (2016) · American Journal of Evaluation · doi:10.1177/1098214016673902 · Cited by 71 · see citing works
Self-citationUnifying Configurational Comparative Methods
Thiem, A. (2013) · Sociological Methods & Research · doi:10.1177/0049124113500481 · Cited by 65 · see citing works
Self-citationDetecting Causal Chains in Small- n Data
Baumgärtner, M. (2012) · Field Methods · doi:10.1177/1525822x12462527 · Cited by 61 · see citing works
Navigating the Complexities of Qualitative Comparative Analysis
Thiem, A. (2014) · Evaluation Review · doi:10.1177/0193841x14550863 · Cited by 58 · see citing works
Self-citationIdentifying Complex Causal Dependencies in Configurational Data with Coincidence Analysis
Baumgärtner, M., & Thiem, A. (2015) · The R Journal · doi:10.32614/rj-2015-014 · Cited by 55 · see citing works
Self-citationEnhancing Sensitivity Diagnostics for Qualitative Comparative Analysis: A Combinatorial Approach
Thiem, A., Spöhel, R., & Duşa, A. (2015) · Political Analysis · doi:10.1093/pan/mpv028 · Cited by 53 · see citing works
Self-citationThiem, A. (2013) · Field Methods · doi:10.1177/1525822x13478135 · Cited by 51 · see citing works
Self-citationRagin, C. C. (2014) · Sociological Methodology · doi:10.1177/0081175014542081 · Cited by 49 · see citing works
Thiem, A. (2016) · Political Analysis · doi:10.1093/pan/mpw024 · Cited by 46 · see citing works
Self-citationThe Scope of General Theory: Methods for Linking Deductive and Inductive Comparative History
BOSWELL, T., & BROWN, C. (1999) · Sociological Methods & Research · doi:10.1177/0049124199028002002 · Cited by 45 · see citing works
A Coincidence Analysis of a Causal Chain
Baumgärtner, M., & Epple, R. (2013) · Sociological Methods & Research · doi:10.1177/0049124113502948 · Cited by 38 · see citing works
Thiem, A., & Baumgärtner, M. (2016) · Comparative Political Studies · doi:10.1177/0010414015626455 · Cited by 36 · see citing works
Self-citationParameters of fit and intermediate solutions in multi-value Qualitative Comparative Analysis
Thiem, A. (2014) · Quality & Quantity · doi:10.1007/s11135-014-0015-x · Cited by 34 · see citing works
Self-citationQualitative Comparative Analysis, Necessary Conditions, and Limited Diversity
Cooper, B., & Glaesser, J. (2015) · Field Methods · doi:10.1177/1525822x15598974 · Cited by 33 · see citing works
Collier, D. (2014) · Sociological Methodology · doi:10.1177/0081175014542568 · Cited by 28 · see citing works
Set-Theoretic Comparative Methods: Less Distinctive Than Claimed
Paine, J. (2016) · Comparative Political Studies · doi:10.1177/0010414014564851 · Cited by 28 · see citing works
Why simulations are appropriate for evaluating Qualitative Comparative Analysis
Rohlfing, I. (2015) · Quality & Quantity · doi:10.1007/s11135-015-0251-8 · Cited by 25 · see citing works
Seawright, J. (2014) · Sociological Methodology · doi:10.1177/0081175014542082 · Cited by 25 · see citing works
Fiss, P. C., Marx, A., & Rihoux, B. (2014) · Sociological Methodology · doi:10.1177/0081175014542079 · Cited by 24 · see citing works
Mind the gap: A review of simulation designs for Qualitative Comparative Analysis
Rohlfing, I. (2015) · Research & Politics · doi:10.1177/2053168015623562 · Cited by 23 · see citing works
Olsen, W. K. (2014) · Sociological Methodology · doi:10.1177/0081175014542080 · Cited by 21 · see citing works
Using Qualitative Comparative Analysis for Identifying Causal Chains in Configurational Data
Thiem, A. (2015) · Sociological Methods & Research · doi:10.1177/0049124115589032 · Cited by 19 · see citing works
Self-citationQCApro: Advanced Functionality for Performing and Evaluating Qualitative Comparative Analysis
Thiem, A. (2016) · doi:10.32614/cran.package.qcapro · Cited by 19 · see citing works
Self-citationModeling Causal Irrelevance in Evaluations of Configurational Comparative Methods
Thiem, A., & Baumgärtner, M. (2016) · Sociological Methodology · doi:10.1177/0081175016654736 · Cited by 18 · see citing works
Self-citationAssessing Set-Theoretic Comparative Methods: A Tool for Qualitative Comparativists?
Munck, G. L. (2016) · Comparative Political Studies · doi:10.1177/0010414015626453 · Cited by 17 · see citing works
QCA is of questionable value for policy research
Tanner, S. (2014) · Policy and Society · doi:10.1016/j.polsoc.2014.08.003 · Cited by 16 · see citing works
Vaisey, S. (2014) · Sociological Methodology · doi:10.1177/0081175014542083 · Cited by 16 · see citing works
Experimental Indistinguishability of Causal Structures
Eberhardt, F. (2013) · Philosophy of Science · doi:10.1086/673865 · Cited by 11 · see citing works
Lucas, S. R. (2014) · Sociological Methodology · doi:10.1177/0081175014544105 · Cited by 6 · see citing works
Determinism and the Method of Difference
Hofmann, U., & Baumgärtner, M. (2011) · THEORIA An International Journal for Theory History and Foundations of Science · doi:10.1387/theoria.988 · Cited by 5 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Multi-method evidence for when and how climate-related disasters contribute to armed conflict risk
Ide, T., Brzoska, M., Donges, J. F., & Schleussner, C.-F. (2020) · Global Environmental Change · doi:10.1016/j.gloenvcha.2020.102063 · Cited by 168 · see citing works
Hanckel, B., Petticrew, M., Thomas, J., & Green, J. (2021) · BMC Public Health · doi:10.1186/s12889-021-10926-2 · Cited by 147 · see citing works
Groth, J., Ide, T., Sakdapolrak, P., Kassa, E., & Hermans, K. (2020) · Global Environmental Change · doi:10.1016/j.gloenvcha.2020.102094 · Cited by 46 · see citing works
Qualitative Comparative Analysis: A Mixed-Method Tool for Complex Implementation Questions
Hill, L. G., Cooper, B. R., & Parker, L. A. (2019) · The Journal of Primary Prevention · doi:10.1007/s10935-019-00536-5 · Cited by 33 · see citing works
- Authors
- Baumgartner, M., & Thiem, A.
- Affiliations
- Department of Philosophy, University of Geneva, Geneva, Switzerland
- University of Geneva (CH)
- ORCID
- Thiem, Alrik
- Venue
- Sociological Methods & Research
- Publisher
- SAGE Publications
- Published
- 2017-05-03
- Volume
- 49
- Issue
- 2
- Pages
- 279-311
- ISSN
- 0049-1241 (ISSN-L)
- 1552-8294
- OpenAlex
- W2610265230
- Semantic Scholar
- 92119e3a4a95914617821b68209fe6ea8a42c2bf
- Cited by
- 193
- References
- 71
- Citations by year
- 2026: 13 · 2025: 22 · 2024: 30 · 2023: 22 · 2022: 29 · 2021: 37 · 2020: 10 · 2019: 18 · 2018: 10 · 2017: 2
- FWCI
- 28.00
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Sociological Methods & Research (publishedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- Funders
- Schweizerischer Nationalfonds zur Förderung der Wissenschaftlichen Forschung
- Topics
- Social Sciences › Sociology and Political Science › Qualitative Comparative Analysis Research
- Physical Sciences › Computer Science › Artificial Intelligence › Bayesian Modeling and Causal Inference
- Physical Sciences › Mathematics › Statistics and Probability › Advanced Causal Inference Techniques
- Keywords
- qualitative comparative analysis, causal inference, configurational comparative methods, simulation study, artificial data, methodological evaluation, conservative solutions, intermediate solutions
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
