Cited by 356Advanced Causal Inference TechniquesSurvey Methodology and NonresponseEconomic and Environmental Valuation

Information equivalence in survey experiments

Open access
Dafoe, Allan, Zhang, Baobao, Caughey, Devin·2018·Political Analysis

Abstract

Survey experiments often manipulate the description of attributes in a hypothetical scenario, with the goal of learning about those attributes’ real-world effects. Such inferences rely on an underappreciated assumption: experimental conditions must be information equivalent (IE) with respect to background features of the scenario. IE is often violated because subjects, when presented with information about one attribute, update their beliefs about others too. Labeling a country “a democracy,” for example, affects subjects’ beliefs about the country’s geographic location. When IE is violated, the effect of the manipulation need not correspond to the quantity of interest (the effect of beliefs about the focal attribute). We formally define the IE assumption, relating it to the exclusion restriction in instrumental-variable analysis. We show how to predict IE violations ex ante and diagnose them ex post with placebo tests. We evaluate three strategies for achieving IE. Abstract encouragement is ineffective. Specifying background details reduces imbalance on the specified details and highly correlated details, but not others. Embedding a natural experiment in the scenario can reduce imbalance on all background beliefs, but raises other issues. We illustrate with four survey experiments, focusing on an extension of a prominent study of the democratic peace.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1017/pan.2018.9

Cites

39 of 39 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • When asked to rate 'the Republican Party' and 'the Democratic Party' on feeling thermometers, respondents need to conjure an image of the typical member of the group they are rating.

    “When asked to rate”

    From Measuring affective polarization: Matching concepts to measures (2026) · cites this paper as Dafoe et al., 2018

    The Formal Exposition (section 2; page unavailable) says survey respondents consider scenarios evoked in their minds, and section 7.1 says they may infer unspecified background features. That supports the general idea that respondents fill in details, but the paper does not address feeling-thermometer ratings of political parties or say respondents picture a typical party member. (LLM, conf=0.76)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

2 of the 41 references couldn’t be matched to a known paper.

Research works printed but not linked: 3

  1. Bansak, Kirk, Jens Hainmueller, Daniel J. Hopkins, and Teppei Yamamoto. 2018. Beyond the breaking point? Survey satisficing in conjoint experiments. Political Analysis 26(1):112-119.
  2. Jeff Manza, Fay Lomax Cook, and I. Benjamin. New York: Oxford, pp. 232-250.
  3. Latura, Audrey. 2015. Material and normative factors in womens professional advancement: experimental evidence from a childcare policy intervention. Paper presented at the American Politics Research Workshop, Harvard University, April 28. http://lists.fas.harvard.edu/pipermail/gov3004- list/attachments/20150427/ea95d274/attachment-0001.pdf.

Cited by

Sorted by citations

Authors
Dafoe, A., Zhang, B., & Caughey, D.
Affiliations
  • Yale University (US)
  • Massachusetts Institute of Technology (US)
Corresponding author
Dafoe, Allan
Venue
Political Analysis
Published
2018-08-03
Volume
26
Issue
4
Pages
399-416
ISSN
  • 1047-1987 (ISSN-L)
  • 1476-4989
OpenAlex
W2885361509
Cited by
356
References
41
Citations by year
2026: 38 · 2025: 70 · 2024: 61 · 2023: 54 · 2022: 54 · 2021: 32 · 2020: 24 · 2019: 14 · 2018: 6 · 2017: 3
FWCI
15.82
Type
journal article
Language
en
Open access
Locations
  • Political Analysis (publishedVersion)
  • RePEc: Research Papers in Economics (submittedVersion)
Funders
  • National Science Foundation
  • Yale University
Topics
  • Physical Sciences › Mathematics › Statistics and Probability › Advanced Causal Inference Techniques
  • Social Sciences › Sociology and Political Science › Survey Methodology and Nonresponse
  • Social Sciences › Economics, Econometrics and Finance › Economics and Econometrics › Economic and Environmental Valuation
Keywords
information equivalence, survey experiment, hypothetical scenarios, belief updating, exclusion restriction, placebo tests, background beliefs, natural experiment, democratic peace

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review