How good are your counterfactuals? Assessing quantitative macro-comparative welfare state research with qualitative criteria
The DOI matches its Crossref record; 7 of the 7 references it cites were found in a registry or an index.
Abstract
All causal statements based on historical data — both in qualitative and quantitative social research — rely on counterfactuals. In quantitative research, scholars attempt to arrive at valid counterfactuals by emulating an experimental design. However, because of treatments that are impossible to manipulate and the non-random assignment of data to treatment and control groups, causal statements are often based on invalid counterfactuals. In qualitative research, scholars attempt to arrive at valid counterfactuals by probing the historical and logical consistency of counterfactuals and by acknowledging the interconnectedness of events. Criteria to evaluate counterfactuals have been developed that allow for a discussion of the quality of counterfactuals used in causal statements. In this article, we suggest using these qualitative criteria to evaluate counterfactuals in quantitative macro-comparative welfare state research. We argue that these criteria can help us identify erroneous causal inferences in quantitative research based on historical data.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Designing Social Inquiry: Scientific Inference in Qualitative Research
King, G., Keohane, R. O., & Verba, S. (1994) · Princeton University Press · doi:10.1515/9781400821211 · Cited by 7,225
Alesina, A., Devleeschauwer, A., Easterly, W., Kurlat, S., & Wacziarg, R. (2003) · Journal of Economic Growth · doi:10.1023/a:1024471506938 · Cited by 3,432
Nested Analysis as a Mixed-Method Strategy for Comparative Research
LIEBERMAN, E. S. (2005) · American Political Science Review · doi:10.1017/s0003055405051762 · Cited by 1,515
Counterfactuals and Hypothesis Testing in Political Science
Fearon, J. D. (1991) · World Politics · doi:10.2307/2010470 · Cited by 729 · cited 3×
“The problem of logical consistency is strongly related to the problem of historical consistency (Fearon, 1991: 193; Lebow, 2000: 582–3; Levy, 2008: 635; Tetlock and Belkin, 1996: 23–5). According to Max Weber (1949), good counterfactual cases should require as few changes as possible compared with the real world (‘min…”
BackgroundThe Phantom Menace: Omitted Variable Bias in Econometric Research
Clarke, K. A. (2005) · Conflict Management and Peace Science · doi:10.1080/07388940500339183 · Cited by 484 · cited 1×
“The fact that interpreting regression models with more than three independent variables becomes increasingly complex (Achen, 2002), and the fact that including relevant control variables might not even reduce omitted variable bias (Clarke, 2005), further aggravates the problem.”
BackgroundRethinking Social Inquiry: Diverse Tools, Shared Standards
(2010) · doi:10.5040/9798216385530 · Cited by 405
When Less is More: Selection Problems in Large- N and Small- N Cross-National Comparisons
Ebbinghaus, B. (2005) · International Sociology · doi:10.1177/0268580905052366 · Cited by 201
Cited by
Designing Research With Qualitative Comparative Analysis (QCA): Approaches, Challenges, and Tools
(2017) · doi:10.1177/0049124117729700 · Cited by 380
Qualitative Comparative Analysis Using R: A Beginner's Guide
Oana, I.-E., Schneider, C. Q., & Thomann, E. (2021) · doi:10.1017/9781009006781 · Cited by 224
Case Studies Nested in Fuzzy-set QCA on Sufficiency: Formalizing Case Selection and Causal Inference
Schneider, C. Q., & Rohlfing, I. (2016) · Sociological Methods & Research · doi:10.1177/0049124114532446 · Cited by 147
A Unifying Framework for Causal Analysis in Set-Theoretic Multimethod Research
Rohlfing, I., & Schneider, C. Q. (2018) · Sociological Methods & Research · doi:10.1177/0049124115626170 · Cited by 98
Metadata
- Authors
- Emmenegger, P.
- Venue
- Journal of European Social Policy
- Publisher
- SAGE Publications
- Published
- 2011-10
- Volume
- 21
- Issue
- 4
- Pages
- 365-380
- ISSN
- 0958-9287 (ISSN-L)
- 1461-7269
- OpenAlex
- W2123422412
- Cited by
- 36
- References
- 7
- FWCI
- 10.72
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: How good are your counterfactuals? Assessing quantitative macro-comparative welfare state research with qualitative criteria. https://app.cochat.ai/papers/10.1177/0958928711412222
- The paper
- Emmenegger, P. (2011). How good are your counterfactuals? Assessing quantitative macro-comparative welfare state research with qualitative criteria. Journal of European Social Policy. https://doi.org/10.1177/0958928711412222
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.