Worth Weighting? How to Think About and Use Weights in Survey Experiments
Open accessThe DOI matches its Crossref record; 6 of the 6 references it cites were found in a registry or an index.
Abstract
The popularity of online surveys has increased the prominence of using sampling weights to enhance claims of representativeness. Yet, much uncertainty remains regarding how these weights should be employed in survey experiment analysis: should they be used? If so, which estimators are preferred? We offer practical advice, rooted in the Neyman–Rubin model, for researchers working with survey experimental data. We examine simple, efficient estimators, and give formulas for their biases and variances. We provide simulations that examine these estimators as well as real examples from experiments administered online through YouGov. We find that for examining the existence of population treatment effects using high-quality, broadly representative samples recruited by top online survey firms, sample quantities, which do not rely on weights, are often sufficient. We found that sample average treatment effect (SATE) estimates did not appear to differ substantially from their weighted counterparts, and they avoided the substantial loss of statistical power that accompanies weighting. When precise estimates of population average treatment effects (PATE) are essential, we analytically show poststratifying on survey weights and/or covariates highly correlated with outcomes to be a conservative choice. While we show substantial gains in simulations, we find limited evidence of them in practice.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
A generalization of sampling without replacement from a finite universe
Horvitz, D. G., & Thompson, D. J. (1952) · Journal of the American Statistical Association · doi:10.1080/01621459.1952.10483446 · Cited by 4,293
Evaluating Online Labor Markets for Experimental Research: Amazon.com's Mechanical Turk
Berinsky, A. J., Huber, G. A., & Lenz, G. S. (2012) · Political Analysis · doi:10.1093/pan/mpr057 · Cited by 4,008
Sampling Techniques. 3rd Edition
Cochran, W. G. (1977) · Zenodo · doi:10.5281/zenodo.15338807 · Cited by 1,136
Population-Based Survey Experiments
Mutz, D. C. (2011) · doi:10.1515/9781400840489 · Cited by 674
Generalizing Observational Study Results: Applying Propensity Score Methods to Complex Surveys
DuGoff, E. H., Schuler, M., & Stuart, E. A. (2014) · Health Services Research · doi:10.1111/1475-6773.12090 · Cited by 605
The Logic of the Survey Experiment Reexamined
Gaines, B. J., Kuklinski, J. H., & Quirk, P. J. (2007) · Political Analysis · doi:10.1093/pan/mpl008 · Cited by 553
Cited by
Coppock, A., & McClellan, O. A. (2019) · Research & Politics · doi:10.1177/2053168018822174 · Cited by 987
Lee, S. L., Pearce, E., Ajnakina, O., Johnson, S., Lewis, G., Mann, F., Pitman, A., Solmi, F., Sommerlad, A., Steptoe, A., et al. (2021) · The Lancet Psychiatry · doi:10.1016/s2215-0366(20)30383-7 · Cited by 531
Why voters who value democracy participate in democratic backsliding
Braley, A., Lenz, G. S., Adjodah, D., Rahnama, H., & Pentland, A. (2023) · Nature Human Behaviour · doi:10.1038/s41562-023-01594-w · Cited by 138
“We therefore take a conservative approach by not weighting the samples, one consistent with the evidence from studies on weighting in survey experiments 67 .”
Guess, A. M., Lockett, D., Lyons, B., Montgomery, J. M., Nyhan, B., & Reifler, J. (2020) · Harvard Kennedy School Misinformation Review · doi:10.37016/mr-2020-004 · Cited by 69
Representativeness and Response Validity Across Nine Opt-In Online Samples
Stagnaro, M. N., Druckman, J., Berinsky, A. J., Arechar, A. A., Willer, R., & Rand, D. G. (2024) · doi:10.31234/osf.io/h9j2d · Cited by 68
Miratrix, L. W., Weiss, M. J., & Henderson, B. (2021) · Journal of Research on Educational Effectiveness · doi:10.1080/19345747.2020.1831115 · Cited by 41
Self-citationPersonalizing Moral Reframing in Interpersonal Conversation: A Field Experiment
Kalla, J. L., Levine, A. S., & Broockman, D. E. (2022) · The Journal of Politics · doi:10.1086/716944 · Cited by 37
Political surveys bias self‐reported economic perceptions
Bailey, J. (2022) · Public Opinion Quarterly · doi:10.1093/poq/nfab054 · Cited by 14
Metadata
- Authors
- Miratrix, L. W., Sekhon, J. S., Theodoridis, A. G., & Campos, L. F.
- Venue
- Political Analysis
- Publisher
- Cambridge University Press (CUP)
- Published
- 2018-05-25
- Volume
- 26
- Issue
- 3
- Pages
- 275-291
- ISSN
- 1047-1987 (ISSN-L)
- 1476-4989
- OpenAlex
- W2616279250
- Cited by
- 187
- References
- 6
- FWCI
- 11.09
- Type
- journal article
- Language
- en
- Open access
- bronze
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.