Validating vignette and conjoint survey experiments against real-world behavior
Open accessAbstract
Significance: Little evidence exists on whether preferences about hypothetical choices measured in a survey experiment are driven by the same structural determinants of the actual choices made in the real world. This study answers this question using a natural experiment as a behavioral benchmark. Comparing the results from conjoint and vignette experiments on which attributes of hypothetical immigrants generate support for naturalization with the outcomes of closely corresponding referendums in Switzerland, we find that the effects estimated from the surveys match the effects of the same attributes in the behavioral benchmark remarkably well. We also find that seemingly subtle differences in survey designs can produce significant differences in performance. Overall, the paired conjoint design performs the best.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Studies have shown hypothetical choices in vignettes to have good predictive validity.
“however, studies have shown hypothetical choices in vignettes to have good predictive validity (see e.g., Alexander & Becker, 1978;Hainmueller et al., 2015).”
From Randomized controlled trial of social interaction police training (2020) · cites this paper as Hainmueller et al., 2015
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Evaluating Online Labor Markets for Experimental Research: Amazon.com's Mechanical Turk
Berinsky, A. J., Huber, G. A., & Lenz, G. S. (2012) · Political Analysis · doi:10.1093/pan/mpr057 · Cited by 4,008 · see citing works
Sensitive questions in surveys.
Tourangeau, R., & Yan, T. (2007) · Psychological Bulletin · doi:10.1037/0033-2909.133.5.859 · Cited by 3,057 · see citing works
Self-reports: How the questions shape the answers.
Schwarz, N. (1999) · American Psychologist · doi:10.1037/0003-066x.54.2.93 · Cited by 2,846 · see citing works
Hainmueller, J., Hopkins, D. J., & Yamamoto, T. (2014) · Political Analysis · doi:10.1093/pan/mpt024 · Cited by 1,990 · see citing works
Self-citationDo People Mean What They Say? Implications for Subjective Survey Data
Bertrand, M., & Mullainathan, S. (2001) · American Economic Review · doi:10.1257/aer.91.2.67 · Cited by 1,391 · see citing works
Who Gets a Swiss Passport? A Natural Experiment in Immigrant Discrimination
HAINMUELLER, J., & HANGARTNER, D. (2013) · American Political Science Review · doi:10.1017/s0003055412000494 · Cited by 433 · see citing works
Self-citation
Flagged first, then by citations
Research works printed but not linked: 17
- Green PE, Rao VR (1971) Conjoint measurement for quantifying judgmental data. J Mark Res 8(3):355-363.
- Rossi PH, Anderson AB (1982) The Factorial Survey Approach: An Introduction (Sage Publications, Beverly Hills, CA), pp 15-67.
- Wallander L (2009) 25 years of factorial surveys in sociology: A review. Soc Sci Res 38(3):505-520.
- Alexander CS, Becker HJ (1978) The use of vignettes in survey research. Public Opin Q 42(1):93-104.
- Arnold HJ, Feldman DC (1981) Social desirability response bias in self-report choice situations. Acad Manage J 24(2):377-385.
- Louviere JJ, Hensher DA, Swait JD (2000) Stated Choice Methods: Analysis and Ap- plications (Cambridge Univ Press, Cambridge, United Kingdom).
- Krosnick JA, Judd CM, Wittenbrink B (2014) The Handbook of Attitudes, eds Albarracin D, Johnson BT, Zanna MP (Psychology Press, New York), pp 21-76.
- Neill HR, Cummings RG, Ganderton PT, Harrison GW, McGuckin T (1994) Hypothetical surveys and real economic commitments. Land Econ 70(2):145-154.
- Bernstein R, Chadha A, Montjoy R (2001) Overreporting voting: Why it happens and why it matters. Public Opin Q 65(1):22-44.
- Benítez-Silva H, Buchinsky M, Man Chan H, Cheidvasser S, Rust J (2004) How large is the bias in self-reported disability? J Appl Econ 19(6):649-670.
- Louviere JJ (1988) Conjoint analysis modelling of stated preferences: A review of theory, methods, recent developments and external validity. J Transp Econ Policy 22(1):93-119.
- Natter M, Feurstein M (2002) Real world performance of choice-based conjoint models. Eur J Oper Res 137(2):448-458.
- Hainmueller J (2011) Entropy balancing for causal effects: A multivariate reweighting method to produce balanced samples in observational studies. Polit Anal 20(1):25-46.
- Diamond PA, Hausman JA (1994) Contingent valuation: Is some number better than no number? J Econ Perspect 8(4):45-64.
- Anderson BA, Silver BD (1986) Measurement and mismeasurement of the validity of the self-reported vote. Am J Pol Sci 30(4):771-785.
- Gelman A, King G (1993) Why are american presidential election campaign polls so variable when votes are so predictable? Br J Polit Sci 23(4):409-451.
- Holbrook AL, Green MC, Krosnick JA (2003) Telephone versus face-to-face interviewing of national probability samples with long questionnaires: Comparisons of respondent sat- isficing and social desirability response bias. Public Opin Q 67(1):79-125.
Cited by
Sorted by citations
How economic, humanitarian, and religious concerns shape European attitudes toward asylum seekers
Bansak, K., Hainmueller, J., & Hangartner, D. (2016) · Science · doi:10.1126/science.aag2147 · Cited by 811 · see citing works
Self-citationInformation equivalence in survey experiments
Dafoe, A., Zhang, B., & Caughey, D. (2018) · Political Analysis · doi:10.1017/pan.2018.9 · Cited by 356 · see citing works
Who are the “illegals”? The social construction of illegality in the United States
Flores, R. D., & Schachter, A. (2018) · American Sociological Review · doi:10.1177/0003122418794635 · Cited by 303 · see citing works
Diverse patients’ attitudes towards Artificial Intelligence (AI) in diagnosis
Robertson, C., Woods, A., Bergstrand, K., Findley, J., Balser, C., & Slepian, M. J. (2023) · PLOS Digital Health · doi:10.1371/journal.pdig.0000237 · Cited by 158 · see citing works
Ideology, Not Affect: What Americans Want from Political Representation
Costa, M. (2021) · American Journal of Political Science · doi:10.1111/ajps.12571 · Cited by 118 · see citing works
Randomized controlled trial of social interaction police training
McLean, K., Wolfe, S. E., Rojek, J., Alpert, G. P., & Smith, M. R. (2020) · Criminology & Public Policy · doi:10.1111/1745-9133.12506 · Cited by 100 · see citing works
Arceneaux, K., & Truex, R. (2023) · Perspectives on Politics · doi:10.1017/s1537592722000901 · Cited by 97 · see citing works
“Starting in January 2021, we began including a simple paired conjoint design in the survey (Hainmueller, Hopkins and Yamamoto, 2014; Hainmueller, Hangartner and Yamamoto, 2015; Teele, Kalla and Rosenbluth, 2018), presenting Republican voters with hypothetical primary candidates for an upcoming…”
The Past, Present, and Future of Experimental Methods in the Social Sciences
Mize, T. D., & Manago, B. (2022) · Social Science Research · doi:10.1016/j.ssresearch.2022.102799 · Cited by 76 · see citing works
Ancestry, Color, or Culture? How Whites Racially Classify Others in the U.S.
Schachter, A., Flores, R. D., & Maghbouleh, N. (2021) · American Journal of Sociology · doi:10.1086/714215 · Cited by 74 · see citing works
Nix, J., Pickett, J. T., & Mitchell, R. J. (2019) · Journal of Experimental Criminology · doi:10.1007/s11292-019-09363-4 · Cited by 41 · see citing works
Hamrak, B. (2025) · Electoral Studies · doi:10.1016/j.electstud.2025.102967 · Cited by 5 · see citing works
- Authors
- Hainmueller, J., Hangartner, D., & Yamamoto, T.
- Affiliations
- Stanford University (US)
- University of Zurich (CH)
- Massachusetts Institute of Technology (US)
- Corresponding author
- Yamamoto, Teppei
- Venue
- Proceedings of the National Academy of Sciences
- Publisher
- National Academy of Sciences
- Published
- 2015-02-02
- Volume
- 112
- Issue
- 8
- Pages
- 2395-2400
- ISSN
- 0027-8424 (ISSN-L)
- 1091-6490
- PMC
- PMC4345583
- OpenAlex
- W2166789068
- Cited by
- 1,344
- References
- 6
- Citations by year
- 2026: 149 · 2025: 188 · 2024: 162 · 2023: 132 · 2022: 171 · 2021: 173 · 2020: 132 · 2019: 91 · 2018: 69 · 2017: 41 · 2016: 29 · 2015: 7
- FWCI
- 117.53
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- Proceedings of the National Academy of Sciences (publishedVersion)
- Europe PMC (PubMed Central) (publishedVersion)
- PubMed Central (publishedVersion)
- PubMed (publishedVersion)
- London School of Economics and Political Science Research Online (London School of Economics and Political Science) (submittedVersion)
- London School of Economics and Political Science Theses Online (London School of Economics and Political Science) (submittedVersion)
- DSpace@MIT (Massachusetts Institute of Technology) (submittedVersion, cc-by-nc)
- Zurich Open Repository and Archive (University of Zurich) (submittedVersion)
- Topics
- Social Sciences › Economics, Econometrics and Finance › Economics and Econometrics › Economic and Environmental Valuation
- Social Sciences › Sociology and Political Science › Survey Methodology and Nonresponse
- Social Sciences › Sociology and Political Science › Names, Identity, and Discrimination Research
- Keywords
- vignette method, conjoint analysis, generalizability, stated preference, naturalization, natural experiment
- MeSH terms
- Choice Behavior, Data Collection, Emigration and Immigration, Humans, Switzerland, Reproducibility of Results
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
