Selection in Surveys: Using Randomized Incentives to Detect and Account for Nonresponse Bias
Open accessThe DOI matches its Crossref record; 11 of the 11 references it cites were found in a registry or an index.
Abstract
We show how to use randomized participation incentives to test and account for nonresponse bias in surveys.We first use data from a survey about labor market conditions, linked to fullpopulation administrative data, to provide evidence of large differences in labor market outcomes between participants and nonparticipants, differences which would not be observable to an analyst who only has access to the survey data.These differences persist even after correcting for observable characteristics, raising concerns about nonresponse bias in survey responses.We then use the randomized incentives in our survey to directly test for nonresponse bias, and find strong evidence that the bias is substantial.Next, we apply a range of existing methods that account for nonresponse bias and find they produce bounds (or point estimates) that are either wide or far from the ground truth.We investigate the failure of these methods by taking a closer look at the determinants of participation, finding that the composition of participants changes in opposite directions in response to incentives and reminder emails.We develop a model of participation that allows for two dimensions of unobserved heterogeneity in the participation decision.Applying the model to our data produces bounds (or point estimates) that are narrower and closer to the ground truth than the other methods.Our results highlight the benefits of including randomized participation incentives in surveys.Both the testing procedure and the methods for bias adjustment may be attractive tools for researchers who are able to embed randomized incentives into their survey.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The PRISMA 2020 statement: An updated guideline for reporting systematic reviews
Page, M. J., McKenzie, J. E., Bossuyt, P. M., Boutron, I., Hoffmann, T. C., Mulrow, C. D., Shamseer, L., Tetzlaff, J. M., Akl, E. A., Brennan, S. E., et al. (2021) · BMJ · doi:10.1136/bmj.n71 · Cited by 108,283
MissForest—Non-parametric missing value imputation for mixed-type data
Stekhoven, D. J., & Bühlmann, P. (2012) · Bioinformatics · doi:10.1093/bioinformatics/btr597 · Cited by 6,156
Nonresponse rates and nonresponse bias in household surveys
Groves, R. M. (2006) · Public Opinion Quarterly · doi:10.1093/poq/nfl033 · Cited by 2,420 · cited 3×
“2This possibility is recognized by Groves (2006), who discusses several indirect methods for measuring nonresponse bias; see also the meta-analysis by Groves and Peytcheva (2008).”
BackgroundShadow Prices, Market Wages, and Labor Supply
Heckman, J. (1974) · Econometrica · doi:10.2307/1913937 · Cited by 1,852 · cited 2×
“The latter assumption is the same parameterization used in the Heckman (1974, 1979) selection model.29 In both cases, we obtain point estimates, but they 28Despite having rejected monotonicity without separability, this does not imply that we reject it when combined with separability, because it’s…”
BackgroundMethodThe Growing Importance of Social Skills in the Labor Market*
Deming, D. J. (2017) · The Quarterly Journal of Economics · doi:10.1093/qje/qjx022 · Cited by 1,727
Estimating Marginal Returns to Education
Carneiro, P., Heckman, J. J., & Vytlacil, E. J. (2011) · American Economic Review · doi:10.1257/aer.101.6.2754 · Cited by 612 · cited 1×
“The next two rows of Figure 10 impose the assumption that the MSR is separable in a covariate Xi, as in Carneiro et al. (2011) and Brinch et al. (2017), among others.”
BackgroundPoverty and Economic Decision-Making: Evidence from Changes in Financial Resources at Payday
Carvalho, L. S., Meier, S., & Wang, S. W. (2016) · American Economic Review · doi:10.1257/aer.20140481 · Cited by 506
Wage Comparisons--A Selectivity Bias
Gronau, R. (1974) · Journal of Political Economy · doi:10.1086/260267 · Cited by 451
Human capital and industrialization: Evidence from the age of enlightenment
Squicciarini, M. P., & Voigtländer, N. (2015) · The Quarterly Journal of Economics · doi:10.1093/qje/qjv025 · Cited by 384
Nine Facts about Top Journals in Economics
Card, D., & DellaVigna, S. (2013) · Journal of Economic Literature · doi:10.1257/jel.51.1.144 · Cited by 333
Technology and Big Data Are Changing Economics: Mining Text to Track Methods
Currie, J., Kleven, H., & Zwiers, E. (2020) · AEA Papers and Proceedings · doi:10.1257/pandp.20201058 · Cited by 86
Cited by
Chetty, R., Friedman, J., Stepner, M., & Team, T. O. I. (2020) · National Bureau of Economic Research · doi:10.3386/w27431 · Cited by 697
The unequal adoption of ChatGPT exacerbates existing inequalities among workers
Humlum, A., & Vestergaard, E. (2025) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2414972121 · Cited by 107
Still Waters, Rapid Currents: Early Labor Market Transformation under Generative AI
Humlum, A., & Vestergaard, E. (2025) · doi:10.3386/w33777 · Cited by 68
Large Language Models, Small Labor Market Effects
Humlum, A., & Vestergaard, E. (2025) · doi:10.2139/ssrn.5219933 · Cited by 27
Bursztyn, L., Haaland, I., Röver, N., & Roth, C. (2025) · doi:10.3386/w33920 · Cited by 18
Metadata
- Authors
- Dutz, D., Huitfeldt, I., Lacouture, S., Mogstad, M., Torgovitsky, A., & van Dijk, W.
- Venue
- National Bureau of Economic Research
- Publisher
- National Bureau of Economic Research
- Published
- 2021-12
- OpenAlex
- W4200072060
- Cited by
- 34
- References
- 11
- Type
- report
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.