Inference for Nonprobability Samples
Open accessAbstract
Although selecting a probability sample has been the standard for decades when making inferences from a sample to a finite population, incentives are increasing to use nonprobability samples. In a world of “big data”, large amounts of data are available that are faster and easier to collect than are probability samples. Design-based inference, in which the distribution for inference is generated by the random mechanism used by the sampler, cannot be used for nonprobability samples. One alternative is quasi-randomization in which pseudo-inclusion probabilities are estimated based on covariates available for samples and nonsample units. Another is superpopulation modeling for the analytic variables collected on the sample units in which the model is used to predict values for the nonsample units. We discuss the pros and cons of each approach.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
There is an extensive literature on approaches for making inferences from data collected from nonprobability samples.
“there is an extensive literature on approaches for making inferences from data collected from nonprobability samples 50-52”
From Unrepresentative big surveys significantly overestimated US vaccine uptake (2021) · cites this paper as [51]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Regression shrinkage and selection via the lasso
Tibshirani, R. (1996) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1996.tb02080.x · Cited by 51,482 · see citing works
The central role of the propensity score in observational studies for causal effects
ROSENBAUM, P. R., & RUBIN, D. B. (1983) · Biometrika · doi:10.1093/biomet/70.1.41 · Cited by 31,241 · see citing works
RUBIN, D. B. (1976) · Biometrika · doi:10.1093/biomet/63.3.581 · Cited by 9,925 · see citing works
Rational choice and the structure of the environment.
Simon, H. A. (1956) · Psychological Review · doi:10.1037/h0042769 · Cited by 5,919 · see citing works
Respondent-Driven Sampling: A New Approach to the Study of Hidden Populations
Heckathorn, D. D. (1997) · Social Problems · doi:10.2307/3096941 · Cited by 4,305 · see citing works
Model Assisted Survey Sampling
Särndal, C., Swensson, B., & Wretman, J. H. (1992) · Springer series in statistics · doi:10.1007/978-1-4612-4378-6 · Cited by 3,417 · see citing works
Beta regression for modelling rates and proportions
Ferrari, S., & Cribari-Neto, F. (2004) · Journal of Applied Statistics · doi:10.1080/0266476042000214501 · Cited by 3,055 · see citing works
BART: Bayesian additive regression trees
Chipman, H. A., George, E. I., & McCulloch, R. E. (2010) · The Annals of Applied Statistics · doi:10.1214/09-aoas285 · Cited by 1,875 · see citing works
van der Laan, M. J., Polley, E., & Hubbard, A. (2007) · Statistical Applications in Genetics and Molecular Biology · doi:10.2202/1544-6115.1309 · Cited by 1,472 · see citing works
Bethlehem, J. (2010) · International Statistical Review · doi:10.1111/j.1751-5823.2010.00112.x · Cited by 1,315 · cited 3× · see citing works
“Bethlehem (2010) reviews this issue for web surveys. Vonk, van Ossenbruggen and Willems (2006) report that ethnic minorities and immigrant groups were systematically underrepresented in Dutch panels. They also found that, relative to the general population, the Dutch online panels contained disproportionately more vot…”
BackgroundInfluentialRubin, D. B. (1981) · The Annals of Statistics · doi:10.1214/aos/1176345338 · Cited by 1,178 · see citing works
Neyman, J. (1934) · Journal of the Royal Statistical Society · doi:10.2307/2342192 · Cited by 1,086 · see citing works
Korn, E. L., & Graubard, B. I. (1999) · Wiley eBooks · doi:10.1002/9781118032619 · Cited by 1,052 · see citing works
(2001) · Technometrics · doi:10.1198/tech.2001.s612 · Cited by 990 · see citing works
Computer and Internet Use in the United States: 2003
Day, J. C., Janus, A. L., & Davis, J. T. (2005) · Cited by 978 · see citing works
Neyman, J. (1992) · Springer Series in Statistics · doi:10.1007/978-1-4612-4380-9_12 · Cited by 932 · see citing works
BART: Bayesian Additive Regression Trees
Chipman, H., George, E. I., & McCulloch, R. E. (2006) · ScholarlyCommons (University of Pennsylvania) · Cited by 889 · see citing works
Rubin, D. B. (1979) · Journal of the American Statistical Association · doi:10.1080/01621459.1979.10482513 · Cited by 852 · see citing works
Summary Report of the AAPOR Task Force on Non-probability Sampling
Baker, R., Brick, J. M., Bates, N., Battaglia, M., Couper, M. P., Dever, J. A., Gile, K. J., & Tourangeau, R. (2013) · Journal of Survey Statistics and Methodology · doi:10.1093/jssam/smt008 · Cited by 663 · see citing works
7. Respondent-Driven Sampling: An Assessment of Current Methodology
Gile, K. J., & Handcock, M. S. (2010) · Sociological Methodology · doi:10.1111/j.1467-9531.2010.01223.x · Cited by 615 · see citing works
Research synthesis: AAPOR report on online panels
Prepared for the AAPOR Executive Council by a Task Force operating under the auspices of the AAPOR Standards Committee, W. M. I., Baker, R., Blumberg, S. J., Brick, J. M., Couper, M. P., Courtright, M., Dennis, J. M., Dillman, D., Frankel, M. R., Garland, P., et al. (2010) · Public Opinion Quarterly · doi:10.1093/poq/nfq048 · Cited by 584 · see citing works
Respondent-Driven Sampling: A New Approach to the Study of Hidden Populations
Heckathorn, D. D. (1997) · Social Problems · doi:10.1525/sp.1997.44.2.03x0221m · Cited by 542 · see citing works
The Use of Propensity Scores to Assess the Generalizability of Results from Randomized Trials
Stuart, E. A., Cole, S. R., Bradshaw, C. P., & Leaf, P. J. (2011) · Journal of the Royal Statistical Society Series A: Statistics in Society · doi:10.1111/j.1467-985x.2010.00673.x · Cited by 524 · see citing works
Resampling Inference With Complex Survey Data
Rao, J. N. K., & Wu, C. (1988) · Journal of the American Statistical Association · doi:10.1080/01621459.1988.10478591 · Cited by 482 · see citing works
On finite population sampling theory under certain linear regression models
Royall, R. M. (1970) · Biometrika · doi:10.1093/biomet/57.2.377 · Cited by 421 · see citing works
Forecasting elections with non-representative polls
Wang, W., Rothschild, D., Goel, S., & Gelman, A. (2015) · International Journal of Forecasting · doi:10.1016/j.ijforecast.2014.06.001 · Cited by 371 · see citing works
The Billion Prices Project: Using Online Prices for Measurement and Research
Cavallo, A. F., & C, R. R. B. (2016) · The Journal of Economic Perspectives · doi:10.1257/jep.30.2.151 · Cited by 302 · see citing works
Models for Nonresponse in Sample Surveys
Little, R. J. A. (1982) · Journal of the American Statistical Association · doi:10.1080/01621459.1982.10477792 · Cited by 262 · see citing works
Holt, D., & Smith, T. M. F. (1979) · Journal of the Royal Statistical Society Series A (General) · doi:10.2307/2344652 · Cited by 258 · see citing works
Practical Tools for Designing and Weighting Survey Samples
Valliant, R. L., Dever, J. A., & Kreuter, F. (2013) · Springer eBooks · doi:10.1007/978-1-4614-6449-5 · Cited by 224 · see citing works
Self-citationOn the Validity of Inferences from Non-random Sample
Smith, T. M. F. (1983) · Journal of the Royal Statistical Society Series A (General) · doi:10.2307/2981454 · Cited by 212 · see citing works
Fitting Regression Models to Survey Data
Lumley, T., & Scott, A. (2017) · Statistical Science · doi:10.1214/16-sts605 · Cited by 209 · see citing works
Lee, S., & Valliant, R. L. (2009) · Sociological Methods & Research · doi:10.1177/0049124108329643 · Cited by 203 · see citing works
Self-citationPerils and potentials of self‐selected entry to epidemiological studies and surveys
Keiding, N., & Louis, T. A. (2016) · Journal of the Royal Statistical Society Series A (Statistics in Society) · doi:10.1111/rssa.12136 · Cited by 199 · see citing works
Estimating Propensity Adjustments for Volunteer Web Surveys
Valliant, R. L., & Dever, J. A. (2011) · Sociological Methods & Research · doi:10.1177/0049124110392533 · Cited by 167 · see citing works
Self-citationWhy the 1936 Literary Digest Poll Failed
Squire, P. (1988) · Public Opinion Quarterly · doi:10.1086/269085 · Cited by 161 · cited 1× · see citing works
“In fact, Roosevelt won the election in a landslide, carrying every state except for Maine and Vermont (Squire, 1988).”
BackgroundThe Foundations of Survey Sampling: A Review
Smith, T. M. F. (1976) · Journal of the Royal Statistical Society Series A (General) · doi:10.2307/2345174 · Cited by 154 · see citing works
Report of the AAPOR task force on non-probability sampling
Baker, R., Brick, J. M., Bates, N., Battaglia, M., Couper, M. P., Dever, J. A., Gile, K. J., & Tourangeau, R. (2013) · Cited by 151 · see citing works
Is the Sky Falling? New Technology, Changing Media, and the Future of Surveys
Couper, M. P. (2013) · Survey research methods · doi:10.18148/srm/2013.v7i3.5751 · Cited by 144 · see citing works
Frost, S. D. W., Brouwer, K. C., Cruz, M. A. F., Ramos, R. L., Ramos, M. E., Lozada, R. M., Magis-Rodríguez, C. L., & Strathdee, S. A. (2006) · Journal of Urban Health · doi:10.1007/s11524-006-9104-z · Cited by 140 · see citing works
Household Surveys with Multiplicity
Sirken, M. G. (1970) · Journal of the American Statistical Association · doi:10.1080/01621459.1970.10481077 · Cited by 128 · see citing works
Murphy, J., Link, M. W., Childs, J. H., Tesfaye, C. L., Dean, E., Stern, M., Pasek, J., Cohen, J., Callegaro, M., & Harwood, P. (2014) · Public Opinion Quarterly · doi:10.1093/poq/nfu053 · Cited by 118 · see citing works
Report of the Inquiry into the 2015 British general election opinion polls
Sturgis, P., Baker, N. J., Callegaro, M., et al. (2016) · ePrints Soton (University of Southampton) · Cited by 110 · see citing works
Generalizing from Unrepresentative Experiments: A Stratified Propensity Score Approach
O’Muircheartaigh, C. A., & Hedges, L. V. (2013) · Journal of the Royal Statistical Society Series C (Applied Statistics) · doi:10.1111/rssc.12037 · Cited by 98 · see citing works
Options for Conducting Web Surveys
Schonlau, M., & Couper, M. P. (2017) · Statistical Science · doi:10.1214/16-sts597 · Cited by 95 · see citing works
Internet Surveys: Can Statistical Adjustments Eliminate Coverage Bias?
Dever, J. A., Rafferty, A. P., & Valliant, R. L. (2007) · Survey research methods · doi:10.18148/srm/2008.v2i2.128 · Cited by 91 · see citing works
Self-citationConstruction of Weights in Surveys: A Review
Haziza, D., & Beaumont, J. (2017) · Statistical Science · doi:10.1214/16-sts608 · Cited by 88 · see citing works
A Systematic Statistical Approach to Evaluating Evidence from Observational Studies
Madigan, D., Stang, P., Berlin, J. A., Schuemie, M. J., Overhage, J. M., Suchard, M. A., Dumouchel, B., Hartzema, A. G., & Ryan, P. B. (2013) · Annual Review of Statistics and Its Application · doi:10.1146/annurev-statistics-022513-115645 · Cited by 75 · see citing works
Subject Acquisition for Web-Based Surveys
Alvarez, R. M., Sherman, R. P., & VanBeselaere, C. (2003) · Political Analysis · doi:10.1093/pan/11.1.23 · Cited by 67 · see citing works
Combining Data from Probability and Non- Probability Samples Using Pseudo-Weights
Elliott, M. R. (2009) · Survey Practice · doi:10.29115/sp-2009-0025 · Cited by 61 · see citing works
Self-citationSchonlau, M., van Soest, A., & Kapteyn, A. (2007) · Survey research methods · doi:10.18148/srm/2007.v1i3.70 · Cited by 60 · see citing works
Obtaining Cancer Risk Factor Prevalence Estimates in Small Areas: Combining Data from Two Surveys
Elliott, M. R., & Davis, W. W. (2005) · Journal of the Royal Statistical Society Series C (Applied Statistics) · doi:10.1111/j.1467-9876.2005.05459.x · Cited by 34 · see citing works
Self-citationSchonlau, M., van Soest, A., & Kapteyn, A. (2007) · SSRN Electronic Journal · doi:10.2139/ssrn.1006108 · Cited by 32 · see citing works
Model Averaging Methods for Weight Trimming.
Elliott, M. R. (2008) · PubMed · Cited by 29 · see citing works
Self-citationA comparison of variance estimators for poststratification to estimated control totals
Dever, J. A., & Valliant, R. L. (2010) · Survey methodology · Cited by 29 · see citing works
Self-citationImputation of Business Survey Data
Kovar, J. G., & Whitridge, P. (1995) · Wiley series in probability and statistics · doi:10.1002/9781118150504.ch22 · Cited by 26 · see citing works
Synthetic Multiple-Imputation Procedure for Multistage Complex Samples
Zhou, H., Elliott, M., & Raghunathan, T. (2016) · Journal of Official Statistics · doi:10.1515/jos-2016-0011 · Cited by 23 · see citing works
Self-citationLeBlanc, M. L., & Tibshirani, R. J. (1998) · Journal of Computational and Graphical Statistics · doi:10.1080/10618600.1998.10474786 · Cited by 22 · see citing works
A Two-Step Semiparametric Method to Accommodate Sampling Weights in Multiple Imputation
Zhou, H., Elliott, M. R., & Raghunathan, T. E. (2015) · Biometrics · doi:10.1111/biom.12413 · Cited by 22 · see citing works
Self-citationGeneral Regression Estimation Adjusted for Undercoverage and Estimated Control Totals
Dever, J. A., & Valliant, R. L. (2016) · Journal of Survey Statistics and Methodology · doi:10.1093/jssam/smw001 · Cited by 21 · see citing works
Self-citationGosnell, H. F. (1937) · Public Opinion Quarterly · doi:10.1086/265040 · Cited by 16 · see citing works
Zhou, H., Elliott, M. R., & Raghunathan, T. E. (2016) · Journal of Survey Statistics and Methodology · doi:10.1093/jssam/smv031 · Cited by 16 · see citing works
Self-citationRecruiting an Internet Panel Using Respondent-Driven Sampling
Schonlau, M., Weidmer, B. A., & Kapteyn, A. (2014) · Journal of Official Statistics · doi:10.2478/jos-2014-0018 · Cited by 15 · see citing works
Incorporating Both Randomized and Observational Data into a Single Analysis
Kaizar, E. E. (2015) · Annual Review of Statistics and Its Application · doi:10.1146/annurev-statistics-010814-020249 · Cited by 15 · see citing works
Bayesian Methods for Unit and Item Nonresponse
Little, R. J. A. (2003) · doi:10.1002/0470867205.ch18 · Cited by 11 · see citing works
Using LASSO to Calibrate Non-probability Samples using Probability Samples.
Chen, K.-T. (2016) · Deep Blue (University of Michigan) · Cited by 8 · see citing works
Berzofsky, M., Williams, R. L., & Biemer, P. P. (2009) · Survey Practice · doi:10.29115/sp-2009-0028 · Cited by 6 · see citing works
Flagged first, then by citations
Research works printed but not linked: 6
- BRICK, J. (2015). Compositional model inference. In Proceedings of the Section on Survey Research Methods 299-307. Amer. Statist. Assoc., Alexandria, VA.
- DEVER, J. and VALLIANT, R. (2014). Estimation with non- probability surveys and the question of external validity. In Proceedings of Statistics Canada Symposium 2014. Statistics Canada, Ottawa, ON.
- DEVER, J. and VALLIANT, R. (2016). GREG estimation with un- dercoverage and estimated controls. Journal of Survey Statistics and Methodology 4 289-318.
- KOHUT, A., KEETER, S., DOHERTY, C., DIMOCK, M. and CHRISTIAN, L. (2012). Assessing the representative- ness of public opinion surveys. Available at http://www. people-press.org/2012/05/15/assessing-the-representativeness- of-public-opinion-surveys/. Pew Research Center; accessed 06-November-2016.
- ROYALL, R. (1971). Linear regression models in finite popula- tion sampling theory. In Foundations of Statistical Inference (V. Godambe and D. Sprott, eds.). Holt, Rinehart, and Winston, Toronto.
- VALLIANT, R., DORFMAN, A. H. and ROYALL, R. M. (2000). Finite Population Sampling and Inference: A Prediction Ap- proach. Wiley, New York.
Cited by
Sorted by citations
Unrepresentative big surveys significantly overestimated US vaccine uptake
Bradley, V. C., Kuriwaki, S., Isakov, M., Sejdinovic, D., Meng, X.-L., & Flaxman, S. (2021) · Nature · doi:10.1038/s41586-021-04198-4 · Cited by 248 · see citing works
Heeringa, S. G., & Berglund, P. A. (2020) · doi:10.1101/2020.02.10.942011 · Cited by 247 · see citing works
Integrating Probability and Nonprobability Samples for Survey Inference
Wiśniowski, A., Sakshaug, J. W., Perez Ruiz, D. A., & Blom, A. G. (2020) · Journal of Survey Statistics and Methodology · doi:10.1093/jssam/smz051 · Cited by 102 · see citing works
Yang, S., Kim, J. K., & Song, R. (2020) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/rssb.12354 · Cited by 73 · see citing works
Ferri-García, R., & Rueda, M. D. M. (2020) · PLOS ONE · doi:10.1371/journal.pone.0231500 · Cited by 67 · see citing works
Measures of the Degree of Departure from Ignorable Sample Selection
Little, R. J. A., West, B. T., Boonstra, P. S., & Hu, J. (2020) · Journal of Survey Statistics and Methodology · doi:10.1093/jssam/smz023 · Cited by 47 · see citing works
Isbanner, S., O’Shaughnessy, P., Steel, D., Wilcock, S., & Carter, S. (2022) · Journal of Medical Internet Research · doi:10.2196/37611 · Cited by 45 · see citing works
- Authors
- Elliott, M. R., & Valliant, R.
- Affiliations
- University of Michigan (US)
- Venue
- Statistical Science
- Publisher
- Institute of Mathematical Statistics
- Published
- 2017-05-01
- Volume
- 32
- Issue
- 2
- Pages
- 249-264
- ISSN
- 0883-4237 (ISSN-L)
- 2168-8745
- OpenAlex
- W2612145410
- Cited by
- 331
- References
- 67
- Citations by year
- 2026: 40 · 2025: 44 · 2024: 51 · 2023: 49 · 2022: 41 · 2021: 28 · 2020: 37 · 2019: 23 · 2018: 13 · 2017: 5
- FWCI
- 14.72
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- Statistical Science (publishedVersion)
- Project Euclid (Cornell University) (submittedVersion)
- Topics
- Physical Sciences › Mathematics › Statistics and Probability › Survey Sampling and Estimation Techniques
- Social Sciences › Sociology and Political Science › Survey Methodology and Nonresponse
- Physical Sciences › Mathematics › Statistics and Probability › Census and Population Estimation
- Keywords
- non-probability samples, design-based inference, big data, finite population inference, survey sampling
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
