The impact of non-response bias due to sampling in public health studies: A comparison of voluntary versus mandatory recruitment in a Dutch national survey on adolescent health
Open accessAbstract
BACKGROUND: In public health monitoring of young people it is critical to understand the effects of selective non-response, in particular when a controversial topic is involved like substance abuse or sexual behaviour. Research that is dependent upon voluntary subject participation is particularly vulnerable to sampling bias. As respondents whose participation is hardest to elicit on a voluntary basis are also more likely to report risk behaviour, this potentially leads to underestimation of risk factor prevalence. Inviting adolescents to participate in a home-sent postal survey is a typical voluntary recruitment strategy with high non-response, as opposed to mandatory participation during school time. This study examines the extent to which prevalence estimates of adolescent health-related characteristics are biased due to different sampling methods, and whether this also biases within-subject analyses. METHODS: Cross-sectional datasets collected in 2011 in Twente and IJsselland, two similar and adjacent regions in the Netherlands, were used. In total, 9360 youngsters in a mandatory sample (Twente) and 1952 youngsters in a voluntary sample (IJsselland) participated in the study. To test whether the samples differed on health-related variables, we conducted both univariate and multivariable logistic regression analyses controlling for any demographic difference between the samples. Additional multivariable logistic regressions were conducted to examine moderating effects of sampling method on associations between health-related variables. RESULTS: As expected, females, older individuals, as well as individuals with higher education levels, were over-represented in the voluntary sample, compared to the mandatory sample. Respondents in the voluntary sample tended to smoke less, consume less alcohol (ever, lifetime, and past four weeks), have better mental health, have better subjective health status, have more positive school experiences and have less sexual intercourse than respondents in the mandatory sample. No moderating effects were found for sampling method on associations between variables. CONCLUSIONS: This is one of first studies to provide strong evidence that voluntary recruitment may lead to a strong non-response bias in health-related prevalence estimates in adolescents, as compared to mandatory recruitment. The resulting underestimation in prevalence of health behaviours and well-being measures appeared large, up to a four-fold lower proportion for self-reported alcohol consumption. Correlations between variables, though, appeared to be insensitive to sampling bias.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Healthy individuals are typically more likely to answer self-administered questionnaires than those with health problems, which might bias the results.
“healthy individuals are typically more likely to answer self-administered questionnaires than those with health problems or problems with other aspects of life,46 which might bias the results.”
From Cross-sectional associations of different types of nature exposure with psychotropic, antihypertensive and asthma medication (2023) · cites this paper as [46]
Differences between those who do and do not respond to inquiries about service satisfaction (bias in nonresponse) exist.
“differences between those who do and do not respond to inquiries about service satisfaction (ie, bias in nonresponse [85])”
From eMental Healthcare Technologies for Anxiety and Depression in Childhood and Adolescence: Systematic Review of Studies Reporting Implementation Outcomes (2018) · cites this paper as [85]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Baron, R. M., & Kenny, D. A. (1986) · Journal of Personality and Social Psychology · doi:10.1037//0022-3514.51.6.1173 · Cited by 91,542 · see citing works
Baron, R. M., & Kenny, D. A. (1986) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.51.6.1173 · Cited by 91,532 · see citing works
The Strengths and Difficulties Questionnaire: A research note
Goodman, R. (1997) · Journal of Child Psychology and Psychiatry · doi:10.1111/j.1469-7610.1997.tb01545.x · Cited by 14,908 · see citing works
The World Health Report 2002 - Reducing Risks, Promoting Healthy Life
Available, N. A. N. (2003) · Education for Health · doi:10.1080/1357628031000116808 · Cited by 6,051 · see citing works
Baumeister, R. F., Campbell, J. D., Krueger, J. I., & Vohs, K. D. (2003) · Psychological Science in the Public Interest · doi:10.1111/1529-1006.01431 · Cited by 3,893 · see citing works
Classical and Modern Regression With Applications.
Steinhorst, R. K., & Myers, R. H. (1988) · Journal of the American Statistical Association · doi:10.2307/2288958 · Cited by 3,705 · see citing works
(2006) · PubMed · Cited by 3,378 · see citing works
Mortality prediction with a single general self-rated health question
DeSalvo, K. B., Bloser, N., Reynolds, K., He, J., & Muntner, P. (2005) · Journal of General Internal Medicine · doi:10.1111/j.1525-1497.2005.00291.x · Cited by 2,303 · see citing works
Harris, L. M., & Dillman, D. A. (1979) · Journal of Marketing Research · doi:10.2307/3150885 · Cited by 1,444 · see citing works
When small effects are impressive
Prentice, D. A., & Miller, D. T. (1992) · Psychological Bulletin · doi:10.1037/0033-2909.112.1.160 · Cited by 1,008 · see citing works
The Strengths and Difficulties Questionnaire (SDQ)
Muris, P., Meesters, C., & van den Berg, F. (2003) · European Child & Adolescent Psychiatry · doi:10.1007/s00787-003-0298-2 · Cited by 920 · see citing works
Psychosocial factors related to adolescent smoking: a critical review of the literature
Tyas, S. L., & Pederson, L. L. (1998) · Tobacco Control · doi:10.1136/tc.7.4.409 · Cited by 899 · see citing works
Understanding The Decision to Participate in a Survey
Groves, R. M., Cialdini, R. B., & Couper, M. P. (1992) · Public Opinion Quarterly · doi:10.1086/269338 · Cited by 813 · see citing works
(2008) · PubMed · Cited by 510 · see citing works
Reliability and Consistency in Self-Reports of Drug Use
O’Malley, P. M., Bachman, J. G., & Johnston, L. D. (1983) · International Journal of the Addictions · doi:10.3109/10826088309033049 · Cited by 435 · see citing works
Wortman, C. B., Rosenthal, R., & Rosnow, R. L. (1976) · The American Journal of Psychology · doi:10.2307/1421418 · Cited by 432 · see citing works
Effect Size, Practical Importance, and Social Policy for Children
McCartney, K., & Rosenthal, R. (2000) · Child Development · doi:10.1111/1467-8624.00131 · Cited by 429 · see citing works
Survey non-response in the Netherlands: Effects on prevalence estimates and associations
Van Loon, A. (2003) · Annals of Epidemiology · doi:10.1016/s1047-2797(02)00257-0 · Cited by 423 · see citing works
Psychometric properties of a Swedish version of the "Strengths and Difficulties Questionnaire"
Smedje, H., Broman, J., Hetta, J. E., & von Knorring, A. ‐. (1999) · European Child & Adolescent Psychiatry · doi:10.1007/s007870050086 · Cited by 416 · see citing works
Criqui, M. H., Barrett-Connor, E. L., & Austin, M. A. (1978) · American Journal of Epidemiology · doi:10.1093/oxfordjournals.aje.a112633 · Cited by 387 · see citing works
Adolescent problem behavior: the influence of parents and peers
Ary, D. V., Duncan, T. E., Duncan, S. C., & Hops, H. (1999) · Behaviour Research and Therapy · doi:10.1016/s0005-7967(98)00133-8 · Cited by 383 · see citing works
Smoking and Drinking: A Review of the Literature
Bien, T. H., & Burge, R. (1990) · International Journal of the Addictions · doi:10.3109/10826089009056229 · Cited by 343 · see citing works
Advances in strategies for minimizing and adjusting for survey nonresponse
Kessler, R. C., Little, R. J. A., & Groves, R. M. (1995) · Epidemiologic Reviews · doi:10.1093/oxfordjournals.epirev.a036176 · Cited by 341 · see citing works
Recruitment Procedures of EPIC-Germany
Boeing, H., Korfmann, A., & Bergmann, M. M. (1999) · Annals of Nutrition and Metabolism · doi:10.1159/000012787 · Cited by 303 · see citing works
Hall, W. (2001) · Nicotine & Tobacco Research · doi:10.1080/14622200110050457 · Cited by 257 · see citing works
Non-response bias in a lifestyle survey
Hill, A., Roberts, J., Ewings, P., & Gunnell, D. (1997) · Journal of Public Health · doi:10.1093/oxfordjournals.pubmed.a024610 · Cited by 220 · see citing works
Caldwell, T. M., Rodgers, B., Jorm, A. F., Christensen, H., Jacomb, P. A., Korten, A. E., & Lynskey, M. T. (2002) · Addiction · doi:10.1046/j.1360-0443.2002.00092.x · Cited by 203 · see citing works
Dunne, M., Martin, N., Bailey, J., Heath, A., Bucholz, K., Madden, P., & Statham, D. (1997) · International Journal of Epidemiology · doi:10.1093/ije/26.4.844 · Cited by 188 · see citing works
The Strengths and Difficulties Questionnaire
Truman, J., Robinson, K., Evans, A. L., Smith, D., Cunningham, L., Millward, R., & Minnis, H. (2003) · European Child and Adolescent Psychiatry · doi:10.1007/s00787-003-0303-9 · Cited by 178 · see citing works
NON-RESPONSE BIAS IN A SAMPLE SURVEY ON ALCOHOL CONSUMPTION
Lahaut, V. M. H. C. J. (2002) · Alcohol and Alcoholism · doi:10.1093/alcalc/37.3.256 · Cited by 164 · see citing works
Needle, R. H., McCubbin, H. I., Lorence, J., & Hochhauser, M. (1983) · International Journal of the Addictions · doi:10.3109/10826088309033058 · Cited by 162 · see citing works
On weighting the rates in non‐response weights
Little, R. J. A., & Vartivarian, S. (2003) · Statistics in Medicine · doi:10.1002/sim.1513 · Cited by 159 · see citing works
Patterns of non-response to a mail survey
Macera, C. A., Jackson, K. L., Davis, D. R., Kronenfeld, J. J., & Blair, S. N. (1990) · Journal of Clinical Epidemiology · doi:10.1016/0895-4356(90)90112-3 · Cited by 145 · see citing works
Non–response bias in alcohol and drug population surveys
Zhao, J., Stockwell, T. R., & MACDONALD, S. (2009) · Drug and Alcohol Review · doi:10.1111/j.1465-3362.2009.00077.x · Cited by 143 · see citing works
Annual research review: on the developmental neuropsychology of substance use disorders
J. Conrod, P., & Nikolaou, K. (2016) · Journal of Child Psychology and Psychiatry · doi:10.1111/jcpp.12516 · Cited by 137 · see citing works
RESPONSE BIAS AND RISK RATIOS IN EPIDEMIOLOGIC STUDIES
Criqui, M. H. (1979) · American Journal of Epidemiology · doi:10.1093/oxfordjournals.aje.a112695 · Cited by 133 · see citing works
Berg, N. (2005) · Encyclopedia of Social Measurement · doi:10.1016/b0-12-369398-5/00038-4 · Cited by 132 · see citing works
Power, C., Rodgers, B., & Hope, S. (1998) · The Lancet · doi:10.1016/s0140-6736(98)23937-7 · Cited by 124 · see citing works
Janzon, L., Hanson, B. S., Isacsson, S., Lindell, S. ‐., & Steen, B. (1986) · Journal of Epidemiology & Community Health · doi:10.1136/jech.40.2.174 · Cited by 119 · see citing works
Covey, L. S., Hughes, D. C., Glassman, A. H., Blazer, D. G., & George, L. K. (1994) · Tobacco Control · doi:10.1136/tc.3.3.222 · Cited by 104 · see citing works
Comparison of Early and Late Respondents to a Postal Health Survey Questionnaire
Paganini‐Hill, A., Hsu, G., Chao, A., & Ross, R. K. (1993) · Epidemiology · doi:10.1097/00001648-199307000-00014 · Cited by 98 · see citing works
Socioeconomic Differences in Smoking in an Urban Swedish Population
Boström, G., Hallqvist, J., Haglund, B. J. A., Romelsjö, A., Svanström, L., & Diderichsen, F. (1993) · Scandinavian Journal of Social Medicine · doi:10.1177/140349489302100204 · Cited by 90 · see citing works
Bias due to Non‐response in a Dutch Survey on Alcohol Consumption
Lemmens, P. H. H. M., TAN, E. S., & Knibbe, R. A. (1988) · British Journal of Addiction · doi:10.1111/j.1360-0443.1988.tb00534.x · Cited by 84 · see citing works
Non-response bias in a web-based health behaviour survey of New Zealand tertiary students
Kypri, K., Samaranayaka, A., CONNOR, J. L., Langley, J. D., & Maclennan, B. (2011) · Preventive Medicine · doi:10.1016/j.ypmed.2011.07.017 · Cited by 81 · see citing works
The Tromsø Heart Study: Responders and Non-responders to a Health Questionnaire, Do They Differ?
Jacobsen, B. K., & Thelle, D. S. (1988) · Scandinavian Journal of Social Medicine · doi:10.1177/140349488801600207 · Cited by 80 · see citing works
O’Neill, T. W., Marsden, D., Silman, A. J., & the European Vertebral Osteoporosis Study Group (1995) · Osteoporosis International · doi:10.1007/bf01622254 · Cited by 80 · see citing works
The association of smoking and drinking habits in a community sample.
Craig, T., & Natta, P. A. V. (1977) · Journal of Studies on Alcohol · doi:10.15288/jsa.1977.38.1434 · Cited by 73 · see citing works
Noll, R. B., Zeller, M. H., Vannatta, K. A., et al. (1997) · Journal of Clinical Child Psychology · doi:10.1207/s15374424jccp2601_4 · Cited by 70 · see citing works
Alati, R., Kinner, S. A., Najman, J. M., Fowler, G., Watt, K., & Green, D. (2004) · Alcohol and Alcoholism · doi:10.1093/alcalc/agh080 · Cited by 68 · see citing works
Patterns of co‐morbidity between alcohol use and other substance use in the Australian population
Degenhardt, L. J., & Hall, W. (2003) · Drug and Alcohol Review · doi:10.1080/0959523021000059776 · Cited by 67 · see citing works
Maclennan, B., Kypri, K., Langley, J. D., & Room, R. (2012) · Drug and Alcohol Dependence · doi:10.1016/j.drugalcdep.2012.05.014 · Cited by 63 · see citing works
Nonresponse in a follow-up to a representative telephone survey of adult drinkers.
Wild, T. C., Cunningham, J., & Adlaf, E. (2001) · Journal of Studies on Alcohol · doi:10.15288/jsa.2001.62.257 · Cited by 50 · see citing works
Gerrits, M. H., van den Oord, E. J. C. G., & Voogt, R. (2001) · Journal of Child Psychology and Psychiatry · doi:10.1111/1469-7610.00755 · Cited by 49 · see citing works
Parental Informed Consent and Sample Bias in Grade-School Children
Frame, C. L., & Strauss, C. C. (1987) · Journal of Social and Clinical Psychology · doi:10.1521/jscp.1987.5.2.227 · Cited by 39 · see citing works
Locker, D., Wiggins, R. D., Sittampalam, Y., et al. (1981) · Journal of Epidemiology & Community Health · doi:10.1136/jech.35.3.208 · Cited by 38 · see citing works
Drop-out and Newcomer Bias in a Community Cardiovascular Follow-up Study
Jooste, P. L., Yach, D., Steenkamp, H. J., et al. (1990) · International Journal of Epidemiology · doi:10.1093/ije/19.2.284 · Cited by 34 · see citing works
Electronic monitoring and health promotion: an evaluation of the E-MOVO Web site by adolescents
de Nooijer, J., Veling, M. L., Ambergen, T., de Vries, H., & de Vries, N. K. (2008) · Health Education Research · doi:10.1093/her/cym086 · Cited by 28 · see citing works
de Nooijer, J., & de Vries, N. K. (2006) · Health Promotion International · doi:10.1093/heapro/dal036 · Cited by 24 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Soga, M., Evans, M. J., Tsuchiya, K., & Fukano, Y. (2021) · Ecological Applications · doi:10.1002/eap.2248 · Cited by 389 · see citing works
An investigation of the diet, exercise, sleep, BMI, and health outcomes of autistic adults
Weir, E., Allison, C., Ong, K. K., & Baron-Cohen, S. (2021) · Molecular Autism · doi:10.1186/s13229-021-00441-x · Cited by 58 · see citing works
Wozney, L., McGrath, P. J., Gehring, N. D., Bennett, K., Huguet, A., Hartling, L., Dyson, M. P., Soleimani, A., & Newton, A. S. (2018) · JMIR Mental Health · doi:10.2196/mental.9655 · Cited by 50 · see citing works
Turunen, A. W., Halonen, J., Korpela, K., Ojala, A., Pasanen, T., Siponen, T., Tiittanen, P., Tyrväinen, L., Yli-Tuomi, T., & Lanki, T. (2023) · Occupational and Environmental Medicine · doi:10.1136/oemed-2022-108491 · Cited by 40 · see citing works
Schieve, L. A., Harris, S., Maenner, M. J., Alexander, A., & Dowling, N. F. (2018) · Emerging Themes in Epidemiology · doi:10.1186/s12982-018-0081-y · Cited by 12 · see citing works
- Authors
- Cheung, K. L., ten Klooster, P. M., Smit, C., de Vries, H., & Pieterse, M. E.
- Affiliations
- Maastricht University (NL)
- University of Twente (NL)
- Corresponding author
- Cheung, Kei Long
- Venue
- BMC Public Health
- Published
- 2017-03-23
- Volume
- 17
- Issue
- 1
- Pages
- 276-276
- ISSN
- 1471-2458 (ISSN-L)
- PMC
- PMC5363011
- OpenAlex
- W2599374901
- Cited by
- 421
- References
- 58
- Citations by year
- 2026: 42 · 2025: 35 · 2024: 56 · 2023: 63 · 2022: 77 · 2021: 76 · 2020: 46 · 2019: 15 · 2018: 10 · 2017: 1
- FWCI
- 50.25
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- Locations
- BMC Public Health (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Research Publications (Maastricht University) (publishedVersion, cc-by)
- University of Twente Research Information (submittedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Brunel University Research Archive (BURA) (Brunel University London) (submittedVersion)
- Research Publications (Maastricht University) (submittedVersion, other-oa)
- Topics
- Social Sciences › Sociology and Political Science › Survey Methodology and Nonresponse
- Physical Sciences › Mathematics › Statistics and Probability › Survey Sampling and Estimation Techniques
- Health Sciences › Health Professions › General Health Professions › Health and Medical Studies
- Keywords
- nonresponse bias, sampling bias, voluntary recruitment, adolescent health, adolescent substance use, alcohol consumption, smoking, sexual behavior, mental health, school experience, prevalence estimation, school-based survey
- MeSH terms
- Adolescent Health, Adolescent, Adolescent Behavior, Cross-Sectional Studies, Female, Humans, Male, Netherlands, Surveys and Questionnaires, Risk Factors, Schools, Bias, Prevalence, Logistic Models, Patient Selection, Substance-Related Disorders, Young Adult
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
