The Design of Field Experiments With Survey Outcomes: A Framework for Selecting More Efficient, Robust, and Ethical Designs
Open accessThe DOI matches its Crossref record; 7 of the 7 references it cites were found in a registry or an index.
Abstract
There is increasing interest in experiments where outcomes are measured by surveys and treatments are delivered by a separate mechanism in the real world, such as by mailers, door-to-door canvasses, phone calls, or online ads. However, common designs for such experiments are often prohibitively expensive, vulnerable to bias, and raise ethical concerns. We show how four methodological practices currently uncommon in such experiments have previously undocumented complementarities that can dramatically relax these constraints when at least two are used in combination: (1) online surveys recruited from a defined sampling frame (2) with at least one baseline wave prior to treatment (3) with multiple items combined into an index to measure outcomes and, (4) when possible, a placebo control. We provide a general and extensible framework that allows researchers to determine the most efficient mix of these practices in diverse applications. Two studies then examine how these practices perform empirically. First, we examine the representativeness of online panel respondents recruited from a defined sampling frame and find that their representativeness compares favorably to phone panel respondents. Second, an original experiment successfully implements all four practices in the context of a door-to-door canvassing experiment. We conclude discussing potential extensions.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Hainmueller, J. (2012) · Political Analysis · doi:10.1093/pan/mpr025 · Cited by 6,026
Hainmueller, J., Hopkins, D. J., & Yamamoto, T. (2014) · Political Analysis · doi:10.1093/pan/mpt024 · Cited by 1,990
Beyond baseline and follow-up: The case for more T in experiments
McKenzie, D. (2012) · Journal of Development Economics · doi:10.1016/j.jdeveco.2012.01.002 · Cited by 871
GERBER, A. S., GIMPEL, J. G., GREEN, D. P., & SHAW, D. R. (2011) · American Political Science Review · doi:10.1017/s000305541000047x · Cited by 596 · cited 1×
“…FTF No n/a No Gerber (2004) Phone No No n/a No Gerber, Karlan, and Bergan (2009) Phone No n/a No Gerber, Huber, and Washington (2010) Phone n/a No Gerber et al. (2011) Phone No No n/a Humphreys and Weinstein (2012) FTF No n/a Miller and Robyn (1975) Phone No n/a Rogers and Nickerson (2013)…”
BackgroundAn extension of control group design
Solomon, R. L. (1949) · Psychological Bulletin · doi:10.1037/h0062958 · Cited by 541
Lasso adjustments of treatment effect estimates in randomized experiments
Bloniarz, A., Liu, H., Zhang, C.-H., Sekhon, J. S., & Yu, B. (2016) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1510506113 · Cited by 169 · cited 1×
“Baseline survey •Measures covariates at baseline capable of decreasing sampling variability and allowing theories with predictions for heterogeneous effects to be tested (Gerber and Green 2012; Bloniarz et al. 2016).”
BackgroundSelf-citationDemocratic and Republican physicians provide different care on politicized health issues
Hersh, E. D., & Goldenberg, M. N. (2016) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1606609113 · Cited by 165
Cited by
Metalearners for estimating heterogeneous treatment effects using machine learning
Künzel, S. R., Sekhon, J. S., Bickel, P. J., & Yu, B. (2019) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1804597116 · Cited by 1,056
Self-citationKALLA, J. L., & BROOCKMAN, D. E. (2018) · American Political Science Review · doi:10.1017/s0003055417000363 · Cited by 454
Self-citationKALLA, J. L., & BROOCKMAN, D. E. (2020) · American Political Science Review · doi:10.1017/s0003055419000923 · Cited by 310
Self-citationA field experiment on community policing and police legitimacy
Peyton, K., Sierra-Arévalo, M., & Rand, D. G. (2019) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1910157116 · Cited by 254
Kalla, J. L., & Broockman, D. E. (2023) · American Journal of Political Science · doi:10.1111/ajps.12657 · Cited by 170
Self-citation
Metadata
- Authors
- Broockman, D. E., Kalla, J. L., & Sekhon, J. S.
- Venue
- Political Analysis
- Publisher
- Cambridge University Press (CUP)
- Published
- 2017-09-18
- Volume
- 25
- Issue
- 4
- Pages
- 435-464
- ISSN
- 1047-1987 (ISSN-L)
- 1476-4989
- OpenAlex
- W3121869809
- Cited by
- 79
- References
- 7
- FWCI
- 16.07
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.