QCA is of questionable value for policy research
Open accessAbstract
Qualitative Comparative Analysis (QCA) has been championed as a valuable tool for public policy research. Focusing on the field of policy evaluation, this research note assesses QCA by comparing research that uses this method to studies based on standard practices for quantitative policy analysis. While attention is centrally focused on causal inference, questions of measurement are also addressed. The analysis suggests that QCA adds little value to current methods of policy scholarship, and its contribution in fact falls far short, compared with present-day standard practices. For example, a properly defined “net effects” framework — which is pointedly rejected by QCA — provides valuable insights regarding the causal effects that are a central concern of policy evaluation. By contrast, as an approach to policy analysis, QCA suffers from severe limitations in both its framework and its findings.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Methodological debate on QCA is still thriving on issues such as the reliability of causal inferences and the usefulness of the method for policy decisions.
“methodological debate is still thriving on issues such as the reliability of causal inferences [72], alongside more general critiques of the usefulness of the method for policy decisions (see, for instance, [73]).”
From The use of Qualitative Comparative Analysis (QCA) to address causality in complex systems: a systematic review of research on public health interventions (2021) · cites this paper as [73]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
(2001) · RePEc: Research Papers in Economics · Cited by 27,577 · see citing works
Matching as an econometric evaluation estimator: Evidence from evaluating a job training programme
Heckman, J. J., Ichimura, H., & Todd, P. E. (1997) · The Review of Economic Studies · doi:10.2307/2971733 · Cited by 5,486 · see citing works
Redesigning Social Inquiry: Fuzzy Sets and Beyond
Ragin, C. C. (2008) · Cited by 3,398 · see citing works
Nonparametric Estimation of Average Treatment Effects Under Exogeneity: A Review
Imbens, G. W. (2004) · Review of Economics and Statistics · doi:10.1162/003465304323023651 · Cited by 2,994 · see citing works
Mostly Harmless Econometrics: An Empiricist's Companion
Kmenta, J. (2010) · Business Economics · doi:10.1057/be.2009.37 · Cited by 2,617 · see citing works
Card, D., & Krueger, A. B. (2000) · American Economic Review · doi:10.1257/aer.90.5.1397 · Cited by 2,208 · see citing works
Causal Inference Using Potential Outcomes: Design, Modeling, Decisions
Rubin, D. B. (2005) · Journal of the American Statistical Association · doi:10.1198/016214504000001880 · Cited by 2,173 · see citing works
Measuring the Impacts of Teachers II: Teacher Value-Added and Student Outcomes in Adulthood
Chetty, R., Friedman, J. N., & Rockoff, J. E. (2014) · American Economic Review · doi:10.1257/aer.104.9.2633 · Cited by 1,717 · see citing works
The Oxford Handbook of Public Policy
(2008) · Oxford University Press eBooks · doi:10.1093/oxfordhb/9780199548453.001.0001 · Cited by 1,664 · see citing works
Angrist, J. D., & Pischke, J.-S. (2010) · Journal of Economic Perspectives · doi:10.1257/jep.24.2.3 · Cited by 1,566 · see citing works
Set-Theoretic Methods for the Social Sciences: A Guide to Qualitative Comparative Analysis
Schneider, C. Q., & Wagemann, C. (2012) · Cited by 1,345 · see citing works
The Failure of Input‐based Schooling Policies
Hanushek, E. A. (2003) · The Economic Journal · doi:10.1111/1468-0297.00099 · Cited by 1,245 · see citing works
Identification Problems in the Social Sciences
Manski, C. F. (1993) · Sociological Methodology · doi:10.2307/271005 · Cited by 1,176 · see citing works
Rethinking Social Inquiry: Diverse Tools, Shared StandardsPreprint
Brady, H. E., & Collier, D. (2010) · SSRN Electronic Journal · Cited by 871 · see citing works
The economic value of higher teacher quality
Hanushek, E. A. (2010) · Economics of Education Review · doi:10.1016/j.econedurev.2010.12.006 · Cited by 833 · see citing works
(2005) · Choice Reviews Online · doi:10.5860/choice.43-0465 · Cited by 697 · see citing works
Rethinking Social Inquiry: Diverse Tools, Shared Standards
Collier, D. A., & Brady, H. E. (2004) · eScholarship (California Digital Library) · Cited by 562 · see citing works
Natural Experiments in the Social Sciences: A Design-Based Approach
Dunning, T. (2012) · Thammasat University Digital Collections · Cited by 466 · see citing works
What Mean Impacts Miss: Distributional Effects of Welfare Reform Experiments
Bitler, M., Gelbach, J. B., & Hoynes, H. W. (2006) · American Economic Review · doi:10.1257/aer.96.4.988 · Cited by 427 · see citing works
Rethinking Social Inquiry: Diverse Tools, Shared Standards
(2010) · doi:10.5040/9798216385530 · Cited by 405 · see citing works
Reardon, S. F., Grewal, E. T., Kalogrides, D., & Greenberg, E. H. (2012) · Journal of Policy Analysis and Management · doi:10.1002/pam.21649 · Cited by 276 · see citing works
Angrist, J. D., Dynarski, S., Kane, T. J., Pathak, P. A., & Walters, C. (2010) · Journal of Policy Analysis and Management · doi:10.1002/pam.21647 · Cited by 199 · see citing works
Who Benefits from KIPP?Preprint
Angrist, J. D., Dynarski, S., Kane, T. J., Pathak, P. A., & Walters, C. (2010) · SSRN Electronic Journal · doi:10.2139/ssrn.1842087 · Cited by 136 · see citing works
Long-run Impacts of School Desegregation & School Quality on Adult AttainmentsPreprint
Johnson, R. C. (2011) · RePEc: Research Papers in Economics · Cited by 123 · see citing works
Qualitative Comparative Analysis (QCA) in Public Policy Analysis : An Extensive Review
Rihoux, B., Bol, D., & Rezsöhazy, I. (2011) · Cited by 122 · see citing works
The Limitations of Net-Effects Thinking
Ragin, C. C. (2006) · Innovative Comparative Methods for Policy Analysis · doi:10.1007/0-387-28829-5_2 · Cited by 103 · see citing works
Status versus growth: The distributional effects of school accountability policies
Ladd, H. F., & Lauen, D. L. (2010) · Journal of Policy Analysis and Management · doi:10.1002/pam.20504 · Cited by 93 · see citing works
Junk Food in Schools and Childhood Obesity
Datar, A., & Nicosia, N. (2012) · Journal of Policy Analysis and Management · doi:10.1002/pam.21602 · Cited by 93 · see citing works
Innovative Comparative Methods for Policy Analysis
Rihoux, B., & Grimm, H. (2006) · Springer eBooks · doi:10.1007/0-387-28829-5 · Cited by 92 · see citing works
Learning More from Social Experiments: Evolving Analytic Approaches
Verdugo, R. R. (2006) · Teachers College Record The Voice of Scholarship in Education · doi:10.1177/016146810610800526 · Cited by 85 · see citing works
Dunning, Thad (2012): Natural Experiments in the Social Sciences: A DesignBased Approach. Cambridge:
Patrick, E. E. (2012) · Cited by 63 · see citing works
Hudson, J., & Kühner, S. (2013) · Policy and Society · doi:10.1016/j.polsoc.2013.10.001 · Cited by 43 · see citing works
The Effects of Institutional Design on the Utilization of Evaluation
Balthasar, A. (2006) · Evaluation · doi:10.1177/1356389006069139 · Cited by 41 · see citing works
Warren, J., Wistow, J., & Bambra, C. L. (2013) · Policy and Society · doi:10.1016/j.polsoc.2013.10.002 · Cited by 38 · see citing works
Emergence of Schools of Public Policy: Reflections by a Founding Dean
Allison, G. (2009) · Oxford University Press eBooks · doi:10.1093/oxfordhb/9780199548453.003.0003 · Cited by 34 · see citing works
The Diversity and Causality of Welfare State Reforms Explored with Fuzzy-Sets
Pennings, P. (2005) · Quality & Quantity · doi:10.1007/s11135-004-5006-x · Cited by 22 · see citing works
Sen, B. (2012) · Journal of Policy Analysis and Management · doi:10.1002/pam.21601 · Cited by 21 · see citing works
The Impact of California's Cal-Learn Demonstration Project, Final Report
Mauldon, J., Malvin, J., Stiles, J., Nicosia, N., & Seto, E. Y. (2000) · eScholarship (California Digital Library) · Cited by 21 · see citing works
QCA as a Tool for Realistic Evaluations. The Case of the Swiss Environmental Impact Assessment
Befani, B., & Sager, F. (2006) · Bern Open Repository and Information System (University of Bern) · Cited by 15 · see citing works
Lee, S. S. (2013) · Policy and Society · doi:10.1016/j.polsoc.2013.10.005 · Cited by 13 · see citing works
Flagged first, then by citations
Research works printed but not linked: 4
- Greene, W. H. (2012). Econometric analysis (7th ed.). Saddle River, NJ: Prentice Hall.
- Hudson, J., & Ku ¨hner, S. (2013a). Innovative methods for policy analysis: QCA and fuzzy sets. Policy and Society, 32(4), 279-356.
- Lalonde, R. J. (1986). Evaluating econometric evaluations of training programs with experimental data. American Economic Review, 76(4), 604- 620.
- Rubin, D. B. (2005). Causal inference using potential outcomes. Journal of the American Statistical Association, 100(469), 322-331.
Cited by
Sorted by citations
Often Trusted but Never (Properly) Tested: Evaluating Qualitative Comparative Analysis
Baumgartner, M., & Thiem, A. (2020) · Sociological Methods & Research · doi:10.1177/0049124117701487 · Cited by 193 · see citing works
Hanckel, B., Petticrew, M., Thomas, J., & Green, J. (2021) · BMC Public Health · doi:10.1186/s12889-021-10926-2 · Cited by 147 · see citing works
Hanckel, B., Petticrew, M., Thomas, J., & Green, J. (2019) · Systematic Reviews · doi:10.1186/s13643-019-1159-5 · Cited by 10 · see citing works
- Authors
- Tanner, S.
- Affiliations
- University of California, Berkeley (US)
- Corresponding author
- Tanner, Sean
- Venue
- Policy and Society
- Publisher
- Oxford University Press (OUP)
- Published
- 2014-09-01
- Volume
- 33
- Issue
- 3
- Pages
- 287-298
- ISSN
- 1449-4035 (ISSN-L)
- 1839-3373
- OpenAlex
- W1980014966
- Cited by
- 16
- References
- 40
- Citations by year
- 2026: 1 · 2024: 1 · 2023: 1 · 2022: 2 · 2021: 2 · 2019: 7 · 2017: 1 · 2015: 1
- FWCI
- 2.17
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Licence
- academic.oup.com/journals/pages/open_access/funder_policies/chorus/standard_publication_model (vor)
- Locations
- Policy and Society (publishedVersion)
- SSRN Electronic Journal (submittedVersion)
- Topics
- Social Sciences › Sociology and Political Science › Qualitative Comparative Analysis Research
- Social Sciences › Decision Sciences › Management Science and Operations Research › Evaluation and Performance Assessment
- Social Sciences › Political Science and International Relations › Political Science Research and Education
- Keywords
- qualitative comparative analysis, policy evaluation, causal inference, measurement, quantitative policy analysis, methodological critique
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
